diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index f54cea9..e7126f7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -115,14 +115,19 @@ anything. The analysis is **hazard-aware**: SCC-aware propagation (a change to a in a circular-dependency cluster impacts all co-members, via Tarjan from `forge rank`) and a data-driven threshold derived from PageRank centrality and ledger incident history (`effectiveThreshold = base / (1 + hazard)`). `--basic` reverts to the fixed-threshold -mode. +mode. `forge impact` walks reverse dependents; the pre-action check, the ambient prompt hook +and the Stop gate's repair checklist also walk the empirical refutation's repaired sibling +and forward relations (frozen parameters, `SIBLING`/`FORWARD` in `src/atlas.js`) and tag +every file `reverse`, `sibling` or `forward`, because the reverse-only walk missed the +sibling files that were 94.7% of the refutation's misses. The verdict is **advisory by default** — it reports, it does not block. Set `FORGE_ENFORCE=1` to turn the strongest signals into a hard block: - a **vacuous or underspecified** prompt (preflight finds no actionable intent), - **un-assemblable required context** (the completeness gate cannot cover the edit set), -- a **blast radius over threshold** (default ~25 files). +- a **blast radius over threshold** (default ~25 dependent files; sibling/forward + co-change candidates are named in the reason but not counted). Everything else stays a warning the human can override. @@ -200,8 +205,12 @@ to decide whether an edit is safe to start. Surface: `forge reuse query | mint | Two failure modes this layer exists to kill: **partial work** (code changes without the artifacts that depend on it) and **session amnesia** (the next session re-assumes what this one knew). Instructions raise the _probability_ of correct behavior; deterministic -hooks guarantee a _floor_ — with per-task miss rate `1−p` and gate catch rate `c`, -silent misses fall to `(1−p)(1−c)`, and every layer here is one more `c`. +hooks guarantee a _floor_ — with per-task miss rate `1−p`, silent misses fall to +`(1−p)·P(no check fires | miss)`: `(1−p)(1−c)` for one check with catch rate `c`. A second +check lowers that only where it catches what the first cannot; the product `∏(1−cⱼ)` holds +only if the checks fire independently. The same check repeated at another point (Stop, +pre-commit, CI on the same diff) is nested, so the residual is `(1−p)(1−c_max)` (formal +synthesis §5.3, corrected 2026-09-21). **The completion gate (Stop, `src/gate.js`).** The only Stop-path guard that may answer: `completion-gate.sh` runs synchronously (the lesson-mining `cortex.sh stop` stays @@ -225,7 +234,10 @@ injects: learned lessons, the anchored goal, the handoff snapshot, recent commit uncommitted changes — a fresh session orients on evidence, not priors. **The state/decision stores (`src/handoff.js`, `src/decide.js`).** `state.md` is a -bounded REWRITE (snapshot semantics — loader cost stays O(bound) forever); +bounded REWRITE (snapshot semantics — loader cost stays O(bound) forever). Writer and +loader share ONE budget in one unit (`STATE_BUDGET_BYTES`, 8 KB): the writer keeps rows in +priority order (goal, next, decisions, gotchas, in-progress, done) until the body fits, so +the SessionStart loader never cuts what the handoff wrote; `decisions.md` is append-only ADR-lite with a machine-readable `decision` ledger twin (log semantics — supersede, never edit). Both refuse secrets at write. @@ -318,8 +330,11 @@ completeness score `s(x)` is a **logistic** over its features (concreteness, nam vagueness, a smooth `tanh` length term) instead of an additive rubric with magic coefficients and discontinuous word-count steps — the `sigmoid` bounds it to (0,1) with no clamp, every feature's pull stays attributable, and a labeled bank could refine the weights via `predictor.js`'s -`trainLogistic`. The calibrated prior still lands the paper's own examples where they were -(a bare "make the auth better" ≈ 0.23 → ask; a concrete verifyToken edit ≈ 0.63 → proceed). +`trainLogistic`. The hand-set prior (not fit to data) puts the paper's own examples on the +right side of the 0.6 threshold: a bare "make the auth better" ≈ 0.23 → ask; the concrete +verifyToken edit ≈ 0.88 → proceed. That edit scored ≈ 0.63 when the weights were set, with one +concrete anchor (the filename); since a named code identifier became a second anchor it scores +≈ 0.88, and the weights were not re-fit. **The evidence trail (preflight).** Once a goal is anchored, every prompt appends its graded `driftScore` to the session log; `cusum` (until now test-only math) accumulates @@ -332,8 +347,11 @@ the gate lattice (turn ⊂ commit ⊂ PR): the Stop hook gates the turn and CI's gates the PR, so this runs the SAME registry-derived completeness classifier (`classifyPath` from `gate.js`) plus `hasSecret` over staged added lines at the commit boundary — code staged without its doc/state artifact, or a staged secret, is caught -while the fix is still one `git add` away. Each rung is an independent catch layer, so -the silent-miss probability falls multiplicatively. +while the fix is still one `git add` away. The rungs are **not** independent catch +layers: on the same diff the copies fire together, so they do not multiply the catch rate +and the residual stays `(1−p)(1−c_max)`. This rung adds catches only where it sees what the +Stop hook could not — edits made after the turn ended, a host or session where the Stop +hook never ran, or a session whose one Stop block was already spent. **Deep verification (`src/consensus.js`, `forge verify --deep`).** Where plain `verify` asks one oracle (the tests) plus one heuristic, this runs a table of independent lenses @@ -526,6 +544,7 @@ forgekit/ ledger_store.js # git-native on-disk ledger (.forge/ledger/): sharded claims, append-only evidence/tombstone logs, normal-form verify ledger_bridge.js # legacy-store bridge, dormant by default (ledger-only); `FORGE_LEDGER_ONLY=0` re-enables cortex/recall/brain shadow-writes + idempotent `ledger import` ledger_read.js # ledger-only read path by default (`FORGE_LEDGER_ONLY=0` merges legacy∪ledger instead): cortex lesson/fact injection, `recall list`, brain's AGENTS.md index all see teammate knowledge from `ledger merge` + learn_consolidate.js # bin/learn-consolidate.sh: deterministic consolidation of ~/.claude/skills/learned — merge duplicates, drop only ledger-refuted (dormant/retracted/attic) lessons; no model call reuse.js # proof-carrying artifact cache: fingerprint (MinHash+LSH), exact→near→adapt→miss ladder, atlas revalidation embed.js # optional embeddings tier (ADR-0005): FORGE_EMBED=cmd:|http:, swaps MinHash/Jaccard for cosine in `reuse query`/`ledger query`, disk-cached at .forge/embed-cache.jsonl, silent fallback to MinHash context.js # budgeted context assembly + completeness gate: R(edit) set cover, compression ladder, computed missing-set @@ -601,17 +620,17 @@ from the tree it describes. ```mermaid %%{init: {'theme':'base','themeVariables':{'primaryColor':'#201a15','primaryTextColor':'#f2ede7','primaryBorderColor':'#372c22','lineColor':'#f26430','secondaryColor':'#272019','tertiaryColor':'#171310','edgeLabelBackground':'#201a15','clusterBkg':'#171310','clusterBorder':'#4a3b2e','fontFamily':'ui-sans-serif, system-ui, sans-serif','fontSize':'14px'},'flowchart':{'curve':'basis','padding':10,'nodeSpacing':36,'rankSpacing':44}}}%% flowchart LR - test["test
113 files"] - src["src
98 files"] + test["test
117 files"] + src["src
109 files"] landing["landing
61 files"] research["research
37 files"] global["global
5 files"] - bench["bench
2 files"] + bench["bench
3 files"] scripts["scripts
2 files"] docs["docs
1 file"] examples["examples
1 file"] - test -- 227 --> src - bench -- 7 --> src + test -- 240 --> src + bench -- 8 --> src examples -- 4 --> src test -- 2 --> bench test -- 2 --> global diff --git a/CHANGELOG.md b/CHANGELOG.md index a5d9a15..4d8af25 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,91 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +### Added + +- **Universal router** (`src/router`, `forge route universal|outcome|fit|models`). It recommends a model or a cascade across any provider's models, with no vendor, tier or threshold in code. + - **Model:** multidimensional IRT for who solves what, with correlated failures integrated by Gauss–Hermite quadrature; a log-linear cost model; and a cascade policy with a parameter-free default objective (`match-best-single`) plus `target`, `value` and `budget`. + - **Data:** the models live in `data/models.json` and `.forge/models.json`. A shipped prior is fitted on public SWE-bench Verified runs of 11 models from 7 providers. `route outcome` and `route fit` perform a Bayesian update on the project's own outcomes. + - **Measured** (harness-bench run 4, pre-registered, 350 held-out issues): 76.3% solved at $0.093 per task, against 75.1% at $0.364 for the best single model chosen on dev (non-inferior, 74% cheaper). See docs/UNIVERSAL_ROUTING.md for the limits. + +### Fixed + +- **Binary files no longer trip the commit gate's secret scan.** The staged scan reads every + file with `git diff --text`, and the entropy leg flagged the XMP packet id + (`W5M0MpCehiHzreSzNTczkc9d`, a constant fixed by Adobe's XMP spec) that the XMP packet + wrapper carries inside PDFs, JPEGs and PNGs, so ordinary binary commits were refused. A staged file that git + reports as binary (`--numstat` prints `-`/`-`) and that contains a NUL byte now gets the + credential-format grammars only; a `binary` attribute on a text file does not qualify, so + `.gitattributes` cannot switch the entropy leg off. The XMP packet id is also exempt from the + entropy leg everywhere, like lockfile integrity digests. A `ghp_…` token inside a binary is + still refused, and an unreadable diff still fails closed. + +- **A handoff snapshot is read back whole at session start.** `forge handoff` wrote up to + 150 lines to `.forge/state.md` but the SessionStart loader injected only the first 80, so + rows 81–150 of a valid handoff were silently dropped (the budget mismatch the formal + synthesis's T4 correction names). Writer and loader now share one budget in one unit, + `STATE_BUDGET_BYTES` (8 KB of snapshot body), and the writer keeps rows in priority order + (goal and acceptance criteria, next steps, decisions, gotchas and open assumptions, + in-progress files, then done) until the body fits. A section that lost rows ends with + "(+N more not kept …)". Sections are now written in that priority order. Only a hand-edited + or pre-budget file can still overflow the loader, and then the cut names the file. + +- **The verifyToken example's completeness score matches the code again.** The docs + (ARCHITECTURE.md, GUIDE, the cognitive-substrate README) and the `src/preflight.js` comment + said "Change verifyToken in src/auth.js to require length > 20; update tests" scores ≈ 0.63 + (medium risk), but the code gives 0.878 (low risk). The prior was hand-set when that task + had one concrete anchor (the filename, 0.63); since 2026-09-21 a named code identifier is a + second anchor. The docs now show 0.88 and say why; the weights are unchanged. A test pins the + value and checks that the two example outputs print it. + +### Changed + +- **The everyday blast-radius checks walk sibling and forward relations, tagged.** The + substrate pre-action check (so also the ambient prompt hook and the `FORGE_ENFORCE` gate) + and the Stop gate's repair checklist ran the reverse-only walk that the empirical + refutation measured at recall 0.022, where 94.7% of the misses were sibling files. They now + walk reverse + the paper's repaired sibling and forward relations at the frozen parameters + already in `src/atlas.js`, and every file is tagged with the relation that reached it: + `forge substrate` and the ambient advisory print `path (reverse|sibling|forward)` with a + per-relation count, `--json` adds `impact.fileRelations` and `impact.relationCounts`, and the + Stop gate's block reason lists the untouched co-change candidates. The enforce gate still + counts only dependents toward its 25-file block (the wide walk would put 79 of this repo's + 98 source files over it, against 35 today, at precision about 0.09) and names the other + candidates in its reason; `blastRelations` changes what it counts. A wide walk never relabels + a reverse dependent, so its reverse-tagged set equals the reverse-only answer. Scope + decomposition and lesson matching keep using dependents only. `relations: ["reverse"]` + (`substrateCheck`, `repairReason`) is the explicit reverse-only option; `forge impact` and + `predict_impact` are unchanged (reverse-only unless `--all-relations`). The + `source/substrate.json` impact faculties move from `operational-v1` to + `operational-v2-recall`, with a guarantee that says the frozen parameters were tuned on a + different graph builder and are not held-out validated here. + +- **`bin/learn-consolidate.sh` no longer lets a model prune memory.** It sent every learned + lesson to Haiku with "DROP anything … contradicted" and rewrote `~/.claude/skills/learned` + from the answer, which is pruning by the model's own judgment (the research requires pruning + by ground truth). Consolidation is now deterministic (`src/learn_consolidate.js`): exact and + near-duplicate lessons within a project merge (MinHash Jaccard ≥ 0.7, the ledger's own + consolidation threshold), and a lesson is dropped only when its matching ledger claim in + that project is dormant, retracted or pruned to the attic; a lesson the ledger knows nothing + about is kept. `--repo ` names the ledger (default: the current directory), and + `--dry-run` / `--json` report without writing. Originals are archived first, as before. The + model rewrite remains behind an explicit `--llm` first argument, with "contradicted" removed + from its prompt. + +- **`learn-consolidate.sh --llm` works on macOS.** It wrapped the model call in GNU `timeout`, + which stock macOS does not ship; with stderr discarded the missing command failed silently, + the model was never called, and every run ended in "response too short". The call now uses + `timeout`, else Homebrew's `gtimeout`, else runs unwrapped. + +- **The gate docs no longer claim that repeated gates multiply their catch rates.** The + headers of `src/commit_gate.js` and `src/gate.js`, ARCHITECTURE.md §5 and the Mintlify + verification-gates page said each rung (Stop, pre-commit, CI) was an independent catch + layer, so the silent-miss probability fell multiplicatively. The formal synthesis withdrew + that (§5.3, corrected 2026-09-21): the same classifier run on the same diff fires together, + so the residual is `(1−p)(1−c_max)`, and a later rung adds catches only where it sees what + the earlier one could not (edits after the turn, a host where the Stop hook never ran). + Comments and docs only; no behaviour change. + ## [1.0.0] - 2026-09-22 ### Added diff --git a/README.md b/README.md index 377ea7b..8043649 100644 --- a/README.md +++ b/README.md @@ -232,9 +232,12 @@ from a fresh repository graph. - **Git-native team merge.** Claims and append-only logs merge by set union. The join is property-tested for commutativity, associativity, and idempotence. - **Heuristic impact prediction.** Forgekit builds a regex-derived code graph and walks - reverse dependencies to estimate affected files and tests. It is not conservative: it can - miss affected files (including constructs its parser does not recognize) as well as produce - false positives. + reverse dependencies to estimate affected files and tests; the pre-action check and the + Stop gate's repair checklist also walk the empirical refutation's sibling and forward + relations and tag each file with the relation that reached it. It is not conservative: it + can miss affected files (including constructs its parser does not recognize) as well as + produce false positives, and the sibling/forward files are lower-precision co-change + candidates. - **Budgeted context assembly.** Definitions, direct dependants, sibling tests, and trusted lessons are selected under a token budget. Missing required context becomes a question rather than invented context. @@ -407,7 +410,7 @@ and output live in [`docs/GUIDE.md`](docs/GUIDE.md). | | `forge impact` | hazard-aware blast radius — SCC-aware propagation + data-driven threshold from PageRank centrality and ledger incident history | | | `forge scope` | decompose files into independent clusters (+ coupled files you didn't name) | | | `forge context` | budgeted context assembly + completeness gate — what an edit NEEDS known | -| | `forge route` | recommend the cheapest capable model for a task (+ gateway config) | +| | `forge route` | recommend the cheapest capable model for a task (+ gateway config); `route universal`: any provider's models, lowest expected cost for the success asked for, learned from outcomes | | | `forge verify` | independent verification gate — tests + hallucinated-symbol + provenance (--deep: multi-lens consensus) | | | `forge precommit` | commit-level gate — staged code w/o docs + secret scan (FORGE_COMMIT_GATE=block|warn|0) | | **Memory** | `forge cortex` | self-correcting project memory — status / why | diff --git a/bench/impact_cases.mjs b/bench/impact_cases.mjs index 2a38893..b2202f7 100644 --- a/bench/impact_cases.mjs +++ b/bench/impact_cases.mjs @@ -47,7 +47,7 @@ // - test/ledger.test.js imports { mergeStates } (:14) and calls it // (src/ledger_sync.js:3 also names it in the module header — same file, already labeled.) // -// claimText (src/ledger.js) — 8 files +// claimText (src/ledger.js) — 9 files // - src/ledger.js defines it (:610); sketchOf() (:636), termsOf() (:637) and :880 call it // - src/context.js imports { claimText } (:13) and calls it (:185) // - src/dash.js imports { claimText } (:16) and calls it (:58, :389, :400) @@ -56,6 +56,7 @@ // - src/cli.js dynamic-imports { claimText } (:874, :1644) and calls it // - src/cortex_mcp.js dynamic-imports { claimText } (:91) and calls it (:96, :106) // - test/ledger.test.js imports { claimText } (:8) and calls it +// - src/learn_consolidate.js imports { claimText } (:32) and calls it (:110) // (test/dash.test.js:69 mentions the name only inside an assertion message — a string, // not a reference — so it is NOT labeled as a dependent.) // @@ -114,6 +115,7 @@ export const IMPACT_CASES = [ "src/context.js", "src/dash.js", "src/deja.js", + "src/learn_consolidate.js", "src/ledger_store.js", "src/cli.js", "src/cortex_mcp.js", diff --git a/bench/universal-router/README.md b/bench/universal-router/README.md new file mode 100644 index 0000000..0c9be56 --- /dev/null +++ b/bench/universal-router/README.md @@ -0,0 +1,21 @@ +# Universal router: shipped prior + +`data/router_prior.json` is the fit the universal router uses until a project records its own +outcomes (`forge route outcome`, then `forge route fit`). It was fitted from public per-task +results: + +- **Tasks:** the 500 issues of SWE-bench Verified (dataset revision `78f471b`); the issue text is the task. +- **Runs:** eleven models from seven providers, each run once per issue with the same agent scaffold (mini-SWE-agent 2.0.0), taken from SWE-bench/experiments @ `40f164d` (runs dated 2026-02-17). Each run gives a verified resolved/unresolved outcome and the observed cost. + +To regenerate it, build the input JSON, then run: + +```bash +# harness-bench writes the input from SWE-bench Verified + SWE-bench/experiments: +# python3 -m hbench.cli build universal --swe-exp --forgekit +# python3 -c "from hbench.tracks import universal as U; U.export_for_forgekit(Path('universal_all.json'))" +node bench/universal-router/fit_prior.mjs universal_all.json --out data/router_prior.json +``` + +The fit chooses the latent dimension k and the prior scale by 3-fold cross-validation. That selection is recorded in `selection` inside the file. + +Raw per-task results are not redistributed here, only the fitted parameters and their provenance. diff --git a/bench/universal-router/fit_prior.mjs b/bench/universal-router/fit_prior.mjs new file mode 100644 index 0000000..841bae0 --- /dev/null +++ b/bench/universal-router/fit_prior.mjs @@ -0,0 +1,22 @@ +#!/usr/bin/env node +// Fit the universal router's shipped prior (data/router_prior.json) from public per-task +// results. Input: a JSON file built from SWE-bench Verified and SWE-bench/experiments +// (harness-bench writes it; see bench/universal-router/README.md): +// { source: {...}, tasks: [{id, text}], outcomes: { : { : {resolved, cost} } } } +// +// node bench/universal-router/fit_prior.mjs [--out data/router_prior.json] [--only ] +import { readFileSync, writeFileSync } from "node:fs"; +import { buildPrior } from "../../src/router/prior.js"; + +const args = process.argv.slice(2); +const opt = (n, d) => (args.includes(n) ? args[args.indexOf(n) + 1] : d); +const input = JSON.parse(readFileSync(args[0], "utf8")); +const out = opt("--out", new URL("../../data/router_prior.json", import.meta.url).pathname); +const only = opt("--only") ? new Set(JSON.parse(readFileSync(opt("--only"), "utf8"))) : null; +const t0 = Date.now(); +const prior = buildPrior(input, only); +writeFileSync(out, `${JSON.stringify(prior, null, 2)}\n`); +console.log( + `wrote ${out}: ${prior.models.length} models, ${prior.provenance.tasks} tasks, k=${prior.mirt.k}, ` + + `scale=${prior.selection.chosen.scale}, ${((Date.now() - t0) / 1000).toFixed(1)}s`, +); diff --git a/bin/learn-consolidate.sh b/bin/learn-consolidate.sh old mode 100644 new mode 100755 index 56c2c14..d5e6c65 --- a/bin/learn-consolidate.sh +++ b/bin/learn-consolidate.sh @@ -1,9 +1,37 @@ #!/usr/bin/env bash -# Consolidate accumulated learned lessons: merge duplicates, prune trivia, -# keep only durable rules. Run weekly (manually or via cron). Uses Haiku. -# Fixes the append-only bloat of the session-learning hook. +# Consolidate the learned lessons the opt-in session-learner appends to +# ~/.claude/skills/learned: merge duplicates, and drop a lesson ONLY when the forge +# ledger refutes it (its matching claim is dormant, retracted or pruned to the attic). +# Deterministic, no model call — src/learn_consolidate.js does the work. Run weekly. +# +# learn-consolidate.sh [--dir ] [--repo ]... [--dry-run] [--json] +# +# `--repo` names a project whose .forge/ledger supplies the evidence (default: the current +# directory, when it has a ledger). A lesson with no matching ledger claim is always kept. +# +# `--llm` (explicit opt-in, first argument) runs the old Haiku rewrite instead. It prunes +# by the model's own judgment, which the research this project follows rejects for +# memory (prune by ground truth, not by the model's say-so), so it is never the default. set -uo pipefail +# Resolve symlinks (the script is usually linked onto PATH) to find the package root. +SELF="${BASH_SOURCE[0]}" +while [ -L "$SELF" ]; do + link="$(readlink "$SELF")" + case "$link" in + /*) SELF="$link" ;; + *) SELF="$(dirname "$SELF")/$link" ;; + esac +done +ROOT="$(cd "$(dirname "$SELF")/.." && pwd)" + +if [ "${1:-}" != "--llm" ]; then + command -v node >/dev/null 2>&1 || { echo "node not found on PATH"; exit 1; } + exec node "$ROOT/src/learn_consolidate.js" "$@" +fi +shift + +echo "! --llm: consolidating by model judgment (not ledger evidence); originals are archived first" DIR="$HOME/.claude/skills/learned" command -v claude >/dev/null 2>&1 || { echo "claude CLI not found on PATH"; exit 1; } @@ -18,9 +46,11 @@ mkdir -p "$DIR/archive" ts="$(date +%Y%m%d-%H%M%S)" for f in $inputs; do cp "$f" "$DIR/archive/$(basename "$f").$ts.bak"; done +# Contradiction is deliberately NOT a model decision even here: only ledger evidence +# (the default path) may refute a lesson. prompt="You are consolidating a developer's accumulated learned lessons from AI coding sessions. MERGE duplicates and near-duplicates into one rule. DROP anything -trivial, one-off, session-specific, or contradicted. KEEP only durable, reusable +trivial, one-off, or session-specific. KEEP only durable, reusable rules (project gotchas, error->fix patterns, workflow rules). Group under '## ' headers (use '## General' for cross-project). Each rule = one markdown bullet. Do NOT invent anything — only compress what is given. NEVER @@ -29,8 +59,17 @@ include secrets/tokens/PII. Output only the markdown, no preamble. LESSONS: $all" +# `timeout` is GNU coreutils: stock macOS has none (Homebrew coreutils installs it as +# `gtimeout`). Calling a missing `timeout` failed silently here (stderr is discarded), so +# claude never ran and every --llm run on a Mac ended in "response too short". +limited() { + if command -v timeout >/dev/null 2>&1; then timeout 180 "$@" + elif command -v gtimeout >/dev/null 2>&1; then gtimeout 180 "$@" + else "$@"; fi +} + # Uses your logged-in session (slower startup, but authed). Weekly/cron task. -out="$(printf '%s' "$prompt" | timeout 180 claude -p --model haiku 2>/dev/null)" +out="$(printf '%s' "$prompt" | limited claude -p --model haiku 2>/dev/null)" out="$(printf '%s' "$out" | sed '/^[[:space:]]*$/d')" # Guard: never overwrite/delete on an error or empty/too-short response. diff --git a/data/models.json b/data/models.json new file mode 100644 index 0000000..9085022 --- /dev/null +++ b/data/models.json @@ -0,0 +1,190 @@ +{ + "$comment": "Model registry for the universal router. Data only: add models, prices and provider ids here or in .forge/models.json. Prices are USD per million tokens and are used only for models without observed attempt costs. `providers` maps a provider name (as in providers.js) to the id that provider serves; an empty map means the model is recommended by id but cannot be applied until you add a provider id.", + "models": [ + { + "id": "claude-haiku-4.5", + "label": "Claude Haiku 4.5", + "org": "Anthropic", + "run_model_id": "claude-haiku-4-5-20251001", + "benchmark_run": "20260217_mini-v2.0.0_claude-4-5-haiku-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": 1, + "price_out": 5, + "price_source": "https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-22)", + "providers": { + "anthropic": "claude-haiku-4-5-20251001" + } + }, + { + "id": "claude-sonnet-4.5", + "label": "Claude Sonnet 4.5", + "org": "Anthropic", + "run_model_id": "claude-sonnet-4-5-20250929", + "benchmark_run": "20260217_mini-v2.0.0_claude-4-5-sonnet-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": 3, + "price_out": 15, + "price_source": "https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-22)", + "providers": { + "anthropic": "claude-sonnet-4-5-20250929" + } + }, + { + "id": "claude-opus-4.5", + "label": "Claude Opus 4.5", + "org": "Anthropic", + "run_model_id": "claude-4-5-opus", + "benchmark_run": "20260217_mini-v2.0.0_claude-4-5-opus-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": 5, + "price_out": 25, + "price_source": "https://platform.claude.com/docs/en/about-claude/pricing (checked 2026-09-22)", + "providers": { + "anthropic": "claude-opus-4-5" + } + }, + { + "id": "claude-opus-4.6", + "label": "Claude Opus 4.6", + "org": "Anthropic", + "run_model_id": "claude-opus-4-6", + "benchmark_run": "20260217_mini-v2.0.0_claude-4-6-opus", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": { + "anthropic": "claude-opus-4-6" + } + }, + { + "id": "deepseek-v3.2", + "label": "DeepSeek V3.2", + "org": "DeepSeek", + "run_model_id": "deepseek-v3.2", + "benchmark_run": "20260217_mini-v2.0.0_deepseek-3-2-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "gemini-3-flash", + "label": "Gemini 3 Flash", + "org": "Google DeepMind", + "run_model_id": "gemini-3-flash-preview", + "benchmark_run": "20260217_mini-v2.0.0_gemini-3-flash-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "glm-5", + "label": "GLM 5", + "org": "Z-AI", + "run_model_id": "glm-5", + "benchmark_run": "20260217_mini-v2.0.0_glm-5-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "gpt-5.2", + "label": "GPT 5.2", + "org": "OpenAI", + "run_model_id": "gpt-5-2", + "benchmark_run": "20260217_mini-v2.0.0_gpt-5-2-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "gpt-5-mini", + "label": "GPT 5 mini", + "org": "OpenAI", + "run_model_id": "gpt-5-mini-2025-08-07", + "benchmark_run": "20260217_mini-v2.0.0_gpt-5-mini", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "kimi-k2.5", + "label": "Kimi K2.5", + "org": "Moonshot AI", + "run_model_id": "kimi-k2.5", + "benchmark_run": "20260217_mini-v2.0.0_kimi-k2-5-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "minimax-m2.5", + "label": "MiniMax M2.5", + "org": "MiniMax", + "run_model_id": "minimax-m2.5", + "benchmark_run": "20260217_mini-v2.0.0_minimax-2-5-high", + "evidence": "SWE-bench Verified, mini-SWE-agent 2.0.0, one attempt per task (SWE-bench/experiments@40f164d, runs dated 2026-02-17)", + "price_in": null, + "price_out": null, + "price_source": null, + "providers": {} + }, + { + "id": "claude-sonnet-5", + "label": "Claude Sonnet 5", + "org": "Anthropic", + "run_model_id": null, + "benchmark_run": null, + "evidence": null, + "price_in": 3, + "price_out": 15, + "price_source": "forgekit src/model_tiers.json (pricingVerified 2026-07-17)", + "providers": { + "anthropic": "claude-sonnet-5" + }, + "note": "no public per-task runs in the prior: enters cold (population-mean ability, cost from price) until outcomes are recorded" + }, + { + "id": "claude-opus-4.8", + "label": "Claude Opus 4.8", + "org": "Anthropic", + "run_model_id": null, + "benchmark_run": null, + "evidence": null, + "price_in": 5, + "price_out": 25, + "price_source": "forgekit src/model_tiers.json (pricingVerified 2026-07-17)", + "providers": { + "anthropic": "claude-opus-4-8" + }, + "note": "cold until outcomes are recorded" + }, + { + "id": "claude-fable-5", + "label": "Claude Fable 5", + "org": "Anthropic", + "run_model_id": null, + "benchmark_run": null, + "evidence": null, + "price_in": 10, + "price_out": 50, + "price_source": "forgekit src/model_tiers.json (pricingVerified 2026-07-17)", + "providers": { + "anthropic": "claude-fable-5" + }, + "note": "cold until outcomes are recorded" + } + ] +} \ No newline at end of file diff --git a/data/router_prior.json b/data/router_prior.json new file mode 100644 index 0000000..ed74c2e --- /dev/null +++ b/data/router_prior.json @@ -0,0 +1,265 @@ +{ + "version": 1, + "models": [ + "claude-haiku-4.5", + "claude-opus-4.5", + "claude-opus-4.6", + "claude-sonnet-4.5", + "deepseek-v3.2", + "gemini-3-flash", + "glm-5", + "gpt-5-mini", + "gpt-5.2", + "kimi-k2.5", + "minimax-m2.5" + ], + "features": { + "names": [ + "log_chars", + "log_lines", + "code_fences", + "log_constraints", + "log_steps", + "rubric_knn", + "rubric_score", + "log_files", + "log_fanout", + "log_churn", + "past_mistakes", + "ambiguity" + ], + "mean": [ + 7.062787772579041, + 3.2878803054715684, + 1.1190000000000007, + 0.983066988785852, + 0.06182653266665947, + 0.2883249334784194, + 0.2611751963393547, + 0, + 0, + 0, + 0, + 0 + ], + "std": [ + 0.8417200458615252, + 0.9530457540503706, + 1.7333029163997837, + 0.9800798580583188, + 0.2789095760387337, + 0.19965141825734764, + 0.11687710780681453, + 1, + 1, + 1, + 1, + 1 + ] + }, + "mirt": { + "k": 1, + "a": [ + 2.177858803755454, + 4.063503002402341, + 2.8359347040377507, + 3.3986989106967864, + 2.5626646675266604, + 3.3649044433622177, + 2.5468476514321257, + 0.45157979433268414, + 2.532025999749253, + 2.489473808690017, + 3.1102620827554137 + ], + "w": [ + 0.07070713276101422, + 0.1733322346323156, + 0.2222672533855528, + -0.08134656643250822, + 0.16847366289629023, + -0.855541323999771, + 0.7965984160287538, + 0, + 0, + 0, + 0, + 0 + ], + "L": [ + [ + 5.1911704471943665 + ], + [ + 5.220184004941034 + ], + [ + 3.649997500447575 + ], + [ + 5.99852893744851 + ], + [ + 4.7570847722368645 + ], + [ + 4.449400899896764 + ], + [ + 3.839364815356646 + ], + [ + 2.804764383436117 + ], + [ + 3.8106589710829795 + ], + [ + 4.315707369830679 + ], + [ + 4.045751091860787 + ] + ] + }, + "cost": { + "alpha": [ + -1.2900175533689853, + -0.49152431275772285, + -0.9824225464118961, + -0.5433188627351587, + -0.9991377763653935, + -1.107762151289058, + -0.9880484211079457, + -3.2422984987701424, + -1.0022113078344923, + -2.272123352441629, + -2.9650546991343676 + ], + "beta": [ + 0.023932639055548002, + 0.1304987203901819, + -0.00815546199250494, + 0.013730794119413263, + 0.059450774896442855, + -0.010695224225947497, + -0.020293167249973074, + 0, + 0, + 0, + 0, + 0 + ], + "s2": 0.43787253519418, + "rho": 0, + "kappa": -3.2870748890887924, + "source": [ + "observed", + "observed", + "observed", + "observed", + "observed", + "observed", + "observed", + "observed", + "observed", + "observed", + "observed" + ], + "pricedModels": 3, + "n": 5498 + }, + "selection": { + "chosen": { + "k": 1, + "scale": 4, + "heldOutLogLik": -1752.9128596102591 + }, + "table": [ + { + "k": 1, + "scale": 0.5, + "heldOutLogLik": -1824.7244269606604 + }, + { + "k": 1, + "scale": 1, + "heldOutLogLik": -1772.9470699393087 + }, + { + "k": 1, + "scale": 2, + "heldOutLogLik": -1755.6790297911962 + }, + { + "k": 1, + "scale": 4, + "heldOutLogLik": -1752.9128596102591 + }, + { + "k": 1, + "scale": 8, + "heldOutLogLik": -1753.1403085829472 + }, + { + "k": 2, + "scale": 0.5, + "heldOutLogLik": -1830.3849115902071 + }, + { + "k": 2, + "scale": 1, + "heldOutLogLik": -1793.7513544765607 + }, + { + "k": 2, + "scale": 2, + "heldOutLogLik": -1792.166234344018 + }, + { + "k": 2, + "scale": 4, + "heldOutLogLik": -1791.7523579503713 + }, + { + "k": 2, + "scale": 8, + "heldOutLogLik": -1784.0834221415564 + }, + { + "k": 2, + "scale": 16, + "heldOutLogLik": -1796.2047119381166 + }, + { + "k": 3, + "scale": 0.5, + "heldOutLogLik": -1842.7640146747044 + }, + { + "k": 3, + "scale": 1, + "heldOutLogLik": -1801.7336988486554 + }, + { + "k": 3, + "scale": 2, + "heldOutLogLik": -1775.4902564936888 + }, + { + "k": 3, + "scale": 4, + "heldOutLogLik": -1789.8919516415144 + } + ], + "folds": 3 + }, + "provenance": { + "benchmark": "SWE-bench Verified (rev 78f471b)", + "runs": "SWE-bench/experiments@40f164d, mini-SWE-agent 2.0.0, dated 2026-02-17", + "split": "all", + "tasks": 500, + "outcomes": 5500, + "fittedAt": "2026-09-22T05:36:55.196Z" + } +} diff --git a/docs/GUIDE.md b/docs/GUIDE.md index 8113326..172f6cf 100644 --- a/docs/GUIDE.md +++ b/docs/GUIDE.md @@ -116,17 +116,17 @@ $ forge substrate "Change verifyToken in src/auth.js to require length > 20; upd Forge substrate — pre-action check proceed: yes - assumption: medium risk · completeness 0.63 + assumption: low risk · completeness 0.88 route: Haiku 4.5 (simple) · complexity 0.15 driven by: base cost of any task context: complete — 4 required item(s), 1840/12000 tokens (`forge context` for the assembly) - impact: 3 file(s) predicted - - src/auth.js - - src/login.js - - src/session.js + impact: 3 file(s) predicted — 3 reverse + - src/auth.js (reverse) + - src/login.js (reverse) + - src/session.js (reverse) verify: - review impacted files before editing @@ -136,6 +136,17 @@ Forge substrate — pre-action check It found `login.js` and `session.js` — the two files that import `verifyToken` but you never named. That's the "forgot the coupled file" bug, caught _before_ the edit. +Each impacted file carries the relation that reached it. `reverse` files depend on the +change. The pre-action check (and so the ambient hook and the enforce gate) also walks the +empirical refutation's repaired **sibling** relation — a file that shares a dependency with +the target, like `deserializer.js` beside a changed `serializer.js` when both use +`wire_format.js` — and its **forward** relation, what the target itself depends on. The +reverse-only walk measured recall 0.022 on nine real repositories, and 94.7% of its misses +were siblings. Sibling and forward files are co-change candidates to check, not certain +breaks: on forgekit itself they take the median answer from 15 files to about 80 (precision +0.09). Pass `relations: ["reverse"]` to `substrateCheck` for the old answer; `forge impact` +stays reverse-only unless you add `--all-relations`. + **A vague task — it tells you to ask first:** ```console @@ -282,8 +293,9 @@ impacts all co-members) and a data-driven threshold from PageRank centrality and ledger incident history. `--basic` reverts to the fixed-threshold mode. Run `forge atlas build` first. -By default the walk follows **reverse dependencies only** — the files that actually -reference the target. `--all-relations` additionally walks the empirical refutation's +By default `forge impact` follows **reverse dependencies only** — the files that actually +reference the target. (`forge substrate`, the ambient prompt hook and the Stop gate's repair +checklist walk all three relations and tag each file.) `--all-relations` additionally walks the empirical refutation's repaired **sibling** and **forward** rules at their frozen parameters (a file that shares a dependency with the target, and what the target itself depends on). That is a recall instrument, not an everyday view: on forgekit itself the median answer goes from 15 files @@ -410,9 +422,13 @@ Thursday's session — no more each-session re-assumption of what you're working ### `forge handoff ""` — the bounded session snapshot Session memory is volatile; `.forge/state.md` is the checkpoint that survives. One -command rewrites it (never appends — it stays ≤150 lines forever) with what got done, -what comes next, the gotchas, and any assumptions this session proceeded under (gathered -automatically from the session log, along with in-progress git files): +command rewrites it (never appends) with what got done, what comes next, the gotchas, and +any assumptions this session proceeded under (gathered automatically from the session log, +along with in-progress git files). It stays within one 8 KB budget forever — the same budget +the session-start loader injects, so the next session reads back everything the handoff +wrote. When the rows do not fit, they are kept in priority order (goal and acceptance +criteria, next steps, decisions, gotchas and assumptions, in-progress files, then done) and +each cut section says how many rows it dropped: ```bash forge handoff "built the export endpoint" \ @@ -1207,7 +1223,7 @@ Plain `forge cost` remains the per-day spend view via `ccusage`. | `forge cost` | Real per-day spend (via `ccusage`) + the cost ceiling; `--stages` for the measured report. | | `forge scan ` | Vet a skill/MCP (SKILL.md/.mcp.json) for injection/RCE/exfil before install. | | `forge harden` | Wire the pre-commit gate (gitleaks-if-present + `forge precommit`) + sandbox settings; never clobbers a user-authored hook. | -| `forge precommit` | Commit-level gate rung: staged code with no doc/state artifact → finding (same classifier as the Stop gate) + built-in secret scan over staged added lines. `FORGE_COMMIT_GATE=block` refuses the commit, `warn` (default) prints and allows, `0` disables; a detected secret blocks in every mode. | +| `forge precommit` | Commit-level gate rung: staged code with no doc/state artifact → finding (same classifier as the Stop gate) + built-in secret scan over staged added lines (binary files get the credential-format grammars only, not the entropy leg). `FORGE_COMMIT_GATE=block` refuses the commit, `warn` (default) prints and allows, `0` disables; a detected secret blocks in every mode. | | `forge spec [init\|lock\|check]` | Spec-as-contract drift check. | | `forge brand` | Print the active brand token map. | | `forge lean ""` | Scope-minimality footprint for a task — advisory (the Lean Path as a command). | @@ -1224,7 +1240,11 @@ forge substrate "update verifyToken in src/auth.js" --json "okToProceed": false, "assumption": { "risk": "high", "shouldAsk": true, "questions": ["…"] }, "route": { "tier": "simple", "model": { "name": "Haiku 4.5" } }, - "impact": { "impactedFiles": ["src/auth.js", "src/login.js"] }, + "impact": { + "impactedFiles": ["src/auth.js", "src/login.js"], + "fileRelations": { "src/auth.js": "reverse", "src/login.js": "reverse" }, + "relationCounts": { "reverse": 2 }, + }, "verification": { "checklist": ["npm test", "npm run typecheck"] }, } ``` @@ -1249,7 +1269,7 @@ Forge substrate — pre-action advisory (advisory, never blocks): - Under-specified (high risk). Ask before editing: • What constraints must be respected: performance, dependencies, style, compatibility? - Suggested model: Haiku 4.5 (simple); escalate only on a verifier failure. -- Predicted blast radius (2): login.js, auth.js. Review these before editing. +- Predicted blast radius (2: 2 reverse): auth.js (reverse), login.js (reverse). Review these before editing. - Verify with: review impacted files before editing · run the narrowest affected test first ``` @@ -1412,7 +1432,7 @@ ambient pre-action guard there. | ------------------------------------------- | --------------------------------- | ----------------------------------------------- | | `proceed: ASK FIRST` / `okToProceed: false` | task is under-specified | ask the `clarify` questions, don't guess | | `route` | cheapest capable model | start there; escalate only if a verifier fails | -| `impact` | predicted blast radius | read these files before editing | +| `impact` | predicted blast radius, tagged | read the `reverse` files; check `sibling`/`forward` ones for a needed co-change | | `scope` | independent vs. coupled work | split independent groups into separate sessions | | `memory` | past Cortex lessons for this area | context, not law — tests override it | | `verify` | how to prove it works | run it, show the output, then say "done" | @@ -1481,7 +1501,7 @@ Create `global/crew/.md` with frontmatter. It installs into `~/.claude/age | when the ambient hook speaks | `src/substrate.js` → `substrateContext()` | | the cross-tool rule wording | `source/rules.json` → `substrate` section (then `forge init`) | | opt-in LLM adjudication | `FORGE_LLM=1` (+ `FORGE_LLM_AMBIENT=1` for the hook); config in `source/substrate.json` → `llm` | -| opt-in enforcing gate (halt, don't just advise) | `FORGE_ENFORCE=1` — blocks a no-anchor prompt or a very-large-blast action; `src/substrate.js` → `enforceDecision()`. Off by default. | +| opt-in enforcing gate (halt, don't just advise) | `FORGE_ENFORCE=1` — blocks a no-anchor prompt or a very-large-blast action (counted over dependents; sibling/forward candidates are named, not counted — `blastRelations` changes that); `src/substrate.js` → `enforceDecision()`. Off by default. | | verify test timeout | `FORGE_VERIFY_TIMEOUT_MS` (default 600000) | ### Opt into LLM-assisted judgments diff --git a/docs/UNIVERSAL_ROUTING.md b/docs/UNIVERSAL_ROUTING.md new file mode 100644 index 0000000..749c1b5 --- /dev/null +++ b/docs/UNIVERSAL_ROUTING.md @@ -0,0 +1,101 @@ +# Universal routing + +`forge route universal` recommends a model, or a cascade of models ("try A; if a check fails, try B"), for a task. It works with any provider's models. + +The router code names no vendor, model, tier or threshold, and a test enforces that. Models come from data (`data/models.json`, plus `.forge/models.json` in a project). What each model can do and what it costs is learned from verified outcomes. + +## The model + +**1. Who solves what: multidimensional item response theory.** + +``` +P(model m solves task j | θ_j) = σ( a_m − w·x_j + λ_m·θ_j ), θ_j ~ N(0, I_k) +``` + +| Symbol | Meaning | +|---|---| +| `a_m` | the ability of model m | +| `x_j` | task features (text; repository signals when a repo is present) | +| `w` | learned weights that turn features into a difficulty, so unseen tasks get one | +| `θ_j` | the difficulty the features miss, shared by all models through their loadings `λ_m` | + +The shared `θ_j` is what makes failures correlated: if one model fails a task, others are more likely to fail it too. On the public data, P(Opus 4.5 solves | MiniMax M2.5 failed) is 0.25, against 0.77 unconditionally. + +**Fitting.** +- The fit maximises the marginal posterior, with θ integrated by Gauss–Hermite quadrature. The nodes are computed with Golub–Welsch, not tabulated. +- The optimiser is L-BFGS with analytic gradients. +- Observations can be sparse: each task may have been tried by any subset of models. +- The latent dimension k and the prior scale are chosen by K-fold cross-validated likelihood, and the scale grid expands past its edge while the likelihood improves. + +**2. What an attempt costs.** + +``` +log cost = α_m + β·x + ε, E[cost] = exp(α_m + β·x + s²/2) +``` + +- `α_m` and the shared slope `β` are fitted by least squares on observed attempt costs. +- A model that has prices but no observed attempts takes `α_m` from its price. For models with both, `α − log(blended price)` is close to constant, and the input/output blend is chosen to make it most constant. + +**3. Choosing a cascade.** For a cascade s = (m₁, m₂, …), with node probabilities `P[m][q]` and weights `w_q`: + +``` +P(s solves) = 1 − Σ_q w_q Π_{m∈s} (1 − P[m][q]) +E[cost of s] = Σ_i c_{m_i} · Σ_q w_q Π_{l" --model --pass|--fail --cost ` records a verified result. Only a hash of the task and its features are stored, never the text. +- `forge route fit` refits with the shipped fit as the prior mean. This is a Bayesian update: a few local outcomes barely move it, and many outcomes dominate. +- Cost intercepts are updated with a unit-information prior. +- A model in the registry but not in the fit enters "cold", at the population-mean ability, until outcomes arrive. + +**5. Candidates.** +- `--provider ` limits the candidates to models that provider can serve (the `providers` map in the registry). +- `--provider any` (the default) gives advice across every model. +- The shipped registry gives provider ids only for Anthropic models. Add ids for OpenRouter, a LiteLLM gateway or a native API in `.forge/models.json`. + +## Shipped prior + +`data/router_prior.json` was fitted on public per-task results, and `bench/universal-router/README.md` shows how to regenerate it: + +- **Tasks:** the 500 SWE-bench Verified issues. +- **Runs:** eleven models from seven providers with the same scaffold (mini-SWE-agent 2.0.0), one attempt each, dated February 2026. + - Anthropic: Claude Haiku / Sonnet / Opus 4.5 and Opus 4.6 + - OpenAI: GPT-5.2 and GPT-5 mini + - Google: Gemini 3 Flash + - Moonshot: Kimi K2.5 + - MiniMax: M2.5 + - DeepSeek: V3.2 + - Z-AI: GLM-5 +- **Selection:** cross-validation chose k = 1 at prior scale 4. Scale 2 had been the edge of the first grid, so the grid kept expanding until the held-out likelihood stopped improving. + +## Measured + +harness-bench run 4 is pre-registered. It fits on 150 dev issues and scores 350 held-out issues against each model's real outcome and cost. + +| Policy | Solved | $ per task | +|---|---|---| +| universal router, `match-best-single` | 76.3% | $0.093 | +| best single model chosen on dev (Gemini 3 Flash) | 75.1% | $0.364 | +| always Claude Opus 4.5 | 77.4% | $0.760 | +| universal router, `target:0.9` | 81.4% | $0.260 | + +**Against the best single model:** non-inferior (+1.1 points, CI [−2.0, +4.3]) at 74% lower cost. In 5-fold cross-validation it is +3.2 points (CI [+0.4, +6.2]) at −$0.58 per task. + +**Limits (measured):** +- **Where the gain comes from.** Most of it comes from choosing across providers. On the 150-issue fit the router does not beat a fixed cascade chosen on the same dev data; with 400 training issues its target modes are cheaper than the fixed equivalents. +- **Targets are optimistic.** Predicted cascade success is optimistic by 4 to 6 points on the test split, so `target:p` lands below p. A cross-validated calibration map is the planned fix. +- **One scaffold, text-only features.** The data is one agent scaffold and text-only features, with February 2026 prices. Re-fit on your own outcomes. + +## Relation to `forge route` + +`forge route` (tiered: haiku / sonnet / opus / fable) is unchanged and remains the default for Claude Code model selection. The universal router is opt-in: run `forge route universal`, or set `route.objective` in `.forge/config.json` to choose its default objective. diff --git a/docs/cognitive-substrate/README.md b/docs/cognitive-substrate/README.md index 03891d6..4b15995 100644 --- a/docs/cognitive-substrate/README.md +++ b/docs/cognitive-substrate/README.md @@ -44,7 +44,7 @@ Forge substrate — pre-action advisory (advisory, never blocks): - Under-specified (high risk). Ask before editing: • What constraints must be respected: performance, dependencies, style, or compatibility? - Suggested model: Haiku 4.5 (simple); escalate only on a verifier failure. -- Predicted blast radius (2): invoice.js, math.js. Review these before editing. +- Predicted blast radius (2: 2 reverse): invoice.js (reverse), math.js (reverse). Review these before editing. - Verify with: review impacted files before editing · run the narrowest affected test first ``` @@ -80,18 +80,23 @@ $ forge substrate "make the auth better" $ forge substrate "Change verifyToken in src/auth.js to require length > 20; update tests" proceed: yes - assumption: medium risk · completeness 0.63 + assumption: low risk · completeness 0.88 route: Haiku 4.5 (simple) - impact: 3 file(s) predicted - - src/auth.js - - src/login.js (imports verifyToken — you didn't mention it) - - src/session.js (imports verifyToken — you didn't mention it) + impact: 3 file(s) predicted — 3 reverse + - src/auth.js (reverse) + - src/login.js (reverse) ← imports verifyToken; you didn't mention it + - src/session.js (reverse) ← imports verifyToken; you didn't mention it verify: - run the narrowest affected test first, then the broader suite ``` The second run found the two files that import `verifyToken` but you never named — the -"forgot the coupled file" bug, caught _before_ the edit. Add `--json` for machine-readable +"forgot the coupled file" bug, caught _before_ the edit. Each file is tagged with the +relation that reached it: `reverse` (depends on the change), `sibling` (shares a dependency +with it) or `forward` (the change depends on it). The check walks all three by default +because the reverse-only walk missed the sibling files that were 94.7% of the empirical +refutation's misses; siblings and forward files are co-change candidates, lower precision +than dependents, and the enforce gate counts dependents only. Add `--json` for machine-readable output (see [Use it in a script](#use-it-in-a-script)). --- diff --git a/docs/legacy/PLAYBOOK.md b/docs/legacy/PLAYBOOK.md index 94502a4..1b8cec5 100644 --- a/docs/legacy/PLAYBOOK.md +++ b/docs/legacy/PLAYBOOK.md @@ -17,7 +17,7 @@ graphify install --project # optional: adds a code-graph skill to this re Then in Claude Code, once per big repo: `/graphify .` (builds the graph) and `graphify hook install` (keeps it current on every commit). -Housekeeping: run `claude-learn-consolidate` weekly to dedupe/prune learned lessons. +Housekeeping: run `claude-learn-consolidate` weekly to merge duplicate learned lessons; it drops a lesson only when the forge ledger refutes it (`--repo ` supplies the ledger). Minimalism enforcer is always on via Ponytail (`/ponytail`, `/ponytail-review`). --- diff --git a/docs/legacy/RUN.md b/docs/legacy/RUN.md index c543d51..9d9c889 100644 --- a/docs/legacy/RUN.md +++ b/docs/legacy/RUN.md @@ -44,7 +44,7 @@ echo 'export ENABLE_SESSION_LEARNING=1' >> ~/.zshrc && source ~/.zshrc # alrea claude-init # in a repo: write AGENTS.md + thin CLAUDE.md (auto-detect stack) claude-taste # list per-repo UI taste skills claude-taste minimalist-ui # enable one taste for the current repo -claude-learn-consolidate # merge/dedupe/prune learned lessons (weekly; ~1-2 min) +claude-learn-consolidate # merge duplicate learned lessons; drop only ledger-refuted ones (weekly; no model call; --llm for the old Haiku rewrite) ``` ## 5. Skills / agents (auto-fire, or force with /name) diff --git a/global/tools/cognitive-substrate/references/capability-map.md b/global/tools/cognitive-substrate/references/capability-map.md index e6b9ca9..3d47d87 100644 --- a/global/tools/cognitive-substrate/references/capability-map.md +++ b/global/tools/cognitive-substrate/references/capability-map.md @@ -6,7 +6,7 @@ | Learning | `forge cortex`, ledger oracles | External outcomes (tests, CI, human accept/revert) move claim confidence; model weights do not change. | | Imagination | `forge imagine [--run]`, `forge impact` | Predicted breaks + minimal covering test suite; `--run` dry-runs it in a sandboxed worktree. | | Self-correction | `forge verify`, `forge diagnose` | Tests/builds beat model claims; 3× the same failure signature mints a diagnosis + escalation. | -| Impact-awareness | `forge atlas`, `forge impact` | Known symbols/files and likely dependents are surfaced. | +| Impact-awareness | `forge atlas`, `forge impact`, `forge substrate` | Known symbols/files, likely dependents, and sibling/forward co-change candidates are surfaced, each tagged by relation (`forge impact` alone is reverse-only unless `--all-relations`). | | M1 routing | `forge route` | Transparent model-tier recommendation. | | M2 assumption gate | `forge preflight`, `forge context` | Under-specified tasks return *computed* missing-set questions. | | M3 decomposition | `forge scope` | Import clusters show independent vs coupled files. | diff --git a/global/tools/handoff/SKILL.md b/global/tools/handoff/SKILL.md index 279c966..0d429c8 100644 --- a/global/tools/handoff/SKILL.md +++ b/global/tools/handoff/SKILL.md @@ -6,8 +6,10 @@ description: End-of-session checkpoint. Use when finishing, pausing, or switchin # handoff — persist what this session knows Session memory is volatile; `.forge/state.md` is the committed-brain checkpoint the -SessionStart hook re-injects. Rewritten every time (bounded ≤150 lines), never appended — -the next session reads a snapshot, not an archive. +SessionStart hook re-injects. Rewritten every time, never appended, and bounded by the +same 8 KB budget the SessionStart loader injects — the next session reads back the whole +snapshot, not an archive. When rows do not fit, `done` rows go first and next steps and +gotchas stay. ## When - Ending or pausing a work session, or before a risky context switch. diff --git a/mintlify/concepts/cross-session-memory.mdx b/mintlify/concepts/cross-session-memory.mdx index 99b5865..3df5856 100644 --- a/mintlify/concepts/cross-session-memory.mdx +++ b/mintlify/concepts/cross-session-memory.mdx @@ -68,7 +68,7 @@ Two stores keep knowledge across sessions: | Store | Semantics | | ------------------- | ------------------------------------------------------------------------------------ | -| `.forge/state.md` | A bounded **rewrite** (snapshot) — loader cost stays `O(bound)` forever. | +| `.forge/state.md` | A bounded **rewrite** (snapshot); writer and loader share one 8 KB budget, so it is read back whole. | | `.forge/decisions.md` | Append-only **ADR-lite** (`D-####`) with a machine-readable decision ledger twin. | Both refuse secrets at write. `state.md` is re-injected each session start; diff --git a/mintlify/concepts/pre-action-gate.mdx b/mintlify/concepts/pre-action-gate.mdx index 551b28e..f681cd5 100644 --- a/mintlify/concepts/pre-action-gate.mdx +++ b/mintlify/concepts/pre-action-gate.mdx @@ -61,7 +61,10 @@ on. **Blast radius** — the set of files an edit is predicted to impact, read from the code graph. `forge impact` computes it; the pipeline surfaces it before the model touches -anything. +anything. The pipeline tags each file with the relation that reached it: `reverse` +(depends on the change), `sibling` (shares a dependency with it) or `forward` (the change +depends on it). `forge impact` alone walks reverse dependents unless you pass +`--all-relations`. ```bash forge impact verifyToken # predicted impacted files for a symbol @@ -81,7 +84,7 @@ The verdict is **advisory by default** — it reports, it does not block. Set the completeness gate cannot cover the predicted edit set. - the impacted set exceeds the default ~25-file threshold. + the dependents in the impacted set exceed the default ~25-file threshold. diff --git a/mintlify/concepts/verification-gates.mdx b/mintlify/concepts/verification-gates.mdx index c00b610..159af18 100644 --- a/mintlify/concepts/verification-gates.mdx +++ b/mintlify/concepts/verification-gates.mdx @@ -4,9 +4,11 @@ description: "Independent verification, the hallucinated-symbol flag, spec-as-co --- Nothing is "done" without a check you can run — a test, a build exit code, a screenshot. -Forge's verification gates each add one more catch. With per-task miss rate `1 − p` and a -gate catch rate `c`, silent misses fall to `(1 − p)(1 − c)`, and every gate here is one -more `c`. +With per-task miss rate `1 − p`, silent misses fall to `(1 − p)` times the chance that no +check fires on the miss: `(1 − p)(1 − c)` for one gate with catch rate `c`. A further gate +lowers that only where it catches something the others cannot. The same check repeated at +another point (Stop hook, pre-commit, CI on the same diff) fires together with the first, +so the residual stays `(1 − p)(1 − c_max)` rather than a product. **Verification reduces, does not certify.** Crew verifiers and the hallucinated-symbol diff --git a/package.json b/package.json index 0141bd7..69d83ef 100644 --- a/package.json +++ b/package.json @@ -34,6 +34,7 @@ }, "files": [ "src", + "data", "source", "global", "templates", diff --git a/source/substrate.json b/source/substrate.json index 53e11f8..21c682b 100644 --- a/source/substrate.json +++ b/source/substrate.json @@ -4,9 +4,9 @@ "faculties": [ { "id": "memory", "forge": "recall + cortex", "status": "partial", "guarantee": "facts and lessons are persisted as auditable files; relevance is advisory" }, { "id": "learning", "forge": "cortex lessons", "status": "partial", "guarantee": "external outcomes update lesson confidence; no model weights are changed" }, - { "id": "imagination", "forge": "impact graph", "status": "operational-v1", "guarantee": "reverse dependency traversal predicts possible blast radius" }, + { "id": "imagination", "forge": "impact graph", "status": "operational-v2-recall", "guarantee": "the pre-action check walks reverse dependents plus the empirical refutation's repaired sibling and forward relations (frozen parameters) and tags each file with its relation; a recall instrument (precision about 0.09 on this repo), with parameters frozen on a different graph builder, so not held-out validated here" }, { "id": "self-correction", "forge": "verify + doom-loop guard", "status": "partial", "guarantee": "tests/builds are trusted over model claims" }, - { "id": "impact-awareness", "forge": "atlas + impact", "status": "operational-v1", "guarantee": "known symbols/files and likely dependents are surfaced before edits" } + { "id": "impact-awareness", "forge": "atlas + impact", "status": "operational-v2-recall", "guarantee": "known symbols/files, their dependents, and sibling/forward co-change candidates are surfaced before edits, each tagged by relation; the enforce gate counts dependents only, and `forge impact` stays reverse-only unless --all-relations" } ], "mechanisms": [ { "id": "M1", "name": "complexity-aware routing", "command": "forge route", "status": "solved-with-transparency-layer" }, diff --git a/src/atlas.js b/src/atlas.js index c32a416..7d4a288 100644 --- a/src/atlas.js +++ b/src/atlas.js @@ -1225,13 +1225,58 @@ export const SIBLING = Object.freeze({ }); export const FORWARD = Object.freeze({ maxHops: 2, weight: 0.5 }); export const IMPACT_RELATIONS = Object.freeze(["reverse", "sibling", "forward"]); -/** What `impact()` walks unless a caller asks for more. The sibling/forward rules above are - * the paper's repair and they work — but they are a RECALL instrument: on this repo the - * median answer goes from 15 files to 78 of ~450 (max 196), recall 1.00, precision 0.093. - * An everyday "what does this change touch?" wants the focused answer, and a gate whose - * blast threshold is 25 files would otherwise trip on almost every edit. So the wider walk - * is opt-in: `impact(atlas, f, { relations: IMPACT_RELATIONS })`, or `--all-relations`. */ +/** What a bare `impact()` call (and `forge impact`) walks: the focused "what references + * this?" answer. The sibling/forward rules above are the paper's repair and a RECALL + * instrument: on this repo the median answer goes from 15 files to about 80 of ~450 + * (max ~197), recall 1.00, precision 0.093. The recall-critical callers — the substrate + * pre-action check (and so the ambient prompt hook and the enforce gate), and the Stop + * gate's repair checklist — pass IMPACT_RELATIONS and tag every file with the relation + * that reached it; `relations: DEFAULT_IMPACT_RELATIONS` is their explicit reverse-only + * option, and `forge impact --all-relations` is the CLI's wide walk. */ export const DEFAULT_IMPACT_RELATIONS = Object.freeze(["reverse"]); +/** Relations ranked by the strength of their structural claim: a reverse file DEPENDS on + * the change; an llm-verified one was graph- and grep-confirmed to reference it; a + * sibling shares a dependency with it; a forward file is something the change depends on. */ +export const RELATION_ORDER = Object.freeze(["reverse", "llm-verified", "sibling", "forward"]); +/** The relations that mean "this file depends on the change" — what a blocking count uses. */ +export const DEPENDENT_RELATIONS = Object.freeze(["reverse", "llm-verified"]); +/** Position of a relation in RELATION_ORDER (unknown relations sort last). */ +export const relationRank = (r) => { + const i = RELATION_ORDER.indexOf(r); + return i < 0 ? RELATION_ORDER.length : i; +}; + +/** + * Per-file relation tags over one or more impact() reports: each impacted file gets the + * strongest relation (RELATION_ORDER) any of its items was reached by, so output can say + * WHY a file is listed and a count can be taken per relation. + * @param {{impacted?: {relation?: string, node?: {file?: string}}[]}[]} reports + * @returns {Record} file → relation + */ +export function fileRelations(reports) { + /** @type {Record} */ + const out = {}; + for (const r of reports || []) + for (const x of r?.impacted || []) { + const file = x?.node?.file; + const rel = x?.relation || "reverse"; + if (!file) continue; + if (!(file in out) || relationRank(rel) < relationRank(out[file])) out[file] = rel; + } + return out; +} + +/** + * Files ordered strongest relation first, then by path — the display order every + * relation-aware caller uses so the dependents lead and co-change candidates follow. + * @param {string[]} files + * @param {Record} rels file → relation (from fileRelations) + */ +export function byRelation(files, rels) { + return [...files].sort( + (a, b) => relationRank(rels[a]) - relationRank(rels[b]) || (a < b ? -1 : a > b ? 1 : 0), + ); +} const round4 = (x) => Number(x.toFixed(4)); @@ -1357,11 +1402,14 @@ export function impact( // Sibling/forward items carry `relation` + `relationHops`; `hopDistance` stays the // REVERSE-dependency distance (null here), so "direct dependents" filters keep meaning. + // A reverse dependent keeps its label even when a sibling/forward path scores higher: + // the wide walk ADDS files and never relabels a dependent, so the reverse-tagged set of + // a wide walk is exactly the reverse-only answer (a count over it cannot drift). const offer = (id, confidence, relation, path, edgeKinds) => { const node = nodeById.get(id); if (!node || confidence < threshold) return; const prev = visited.get(id); - if (prev && prev.confidence >= round4(confidence)) return; + if (prev && (prev.relation === "reverse" || prev.confidence >= round4(confidence))) return; visited.set(id, { id, node, diff --git a/src/cli.js b/src/cli.js index 9c60d4a..27d6fb5 100755 --- a/src/cli.js +++ b/src/cli.js @@ -1988,6 +1988,9 @@ HANDLERS.route = async (argv) => { console.log(" and calibrating on real routing outcomes needs data forge does not record"); return; } + if (["universal", "outcome", "fit", "models"].includes(argv[1]) || argv.includes("--universal")) { + return routeUniversalCli(argv); + } const json = argv.includes("--json"); const apply = argv.includes("--apply"); const providerIdx = argv.indexOf("--provider"); @@ -2051,6 +2054,113 @@ HANDLERS.route = async (argv) => { } return; }; +// Universal router (src/router): any provider's models, chosen by expected cost for the success +// probability asked for. Models come from data/models.json and .forge/models.json. +async function routeUniversalCli(argv) { + const U = await import("./router/index.js"); + const { loadRegistry } = await import("./router/registry.js"); + const json = argv.includes("--json"); + const val = (flag) => (argv.includes(flag) ? argv[argv.indexOf(flag) + 1] : undefined); + const VALUED = new Set(["--objective", "--provider", "--model", "--cost", "--depth"]); + const words = argv + .slice(1) + .filter((a, i, arr) => !a.startsWith("--") && !VALUED.has(arr[i - 1] ?? "")); + const sub = ["outcome", "fit", "models", "universal"].includes(words[0]) + ? words.shift() + : "universal"; + const root = process.cwd(); + if (sub === "models") { + const reg = loadRegistry(root); + const fit = U.loadRouterModel(root); + const rows = reg.models.map((m) => ({ + id: m.id, + org: m.org ?? null, + status: fit?.models.includes(m.id) ? "fitted" : "cold", + providers: Object.keys(m.providers ?? {}), + price: m.price_in != null ? `${m.price_in}/${m.price_out}` : null, + })); + if (json) + return console.log( + JSON.stringify({ sources: reg.sources, fit: fit?.origin ?? null, models: rows }, null, 2), + ); + heading(`${BRAND.brand} route models — registry (${reg.sources.join(" + ")})\n`); + for (const r of rows) + console.log( + ` ${r.id.padEnd(22)} ${String(r.org ?? "").padEnd(16)} ${r.status.padEnd(7)} ${r.price ? `$${r.price}/Mtok`.padEnd(14) : "".padEnd(14)} ${r.providers.join(", ") || "(no provider id: advice only)"}`, + ); + console.log(`\n fit in use: ${fit?.origin ?? "none"}`); + return; + } + if (sub === "fit") { + const model = U.fitRouter(root); + if (json) return console.log(JSON.stringify(model.provenance, null, 2)); + console.log( + ` refit on ${model.provenance.local.outcomes} recorded outcome(s) over ${model.provenance.local.tasks} task(s); wrote .forge/router_model.json`, + ); + return; + } + const task = words.join(" "); + if (!task) { + console.error( + 'usage: forge route universal "" [--objective match-best-single|target:

|value:<$>|budget:<$>] [--provider |any] [--depth ] [--json]\n' + + ' forge route outcome "" --model --pass|--fail [--cost ]\n' + + " forge route fit | forge route models", + ); + process.exitCode = 1; + return; + } + if (sub === "outcome") { + const passed = argv.includes("--pass") ? true : argv.includes("--fail") ? false : undefined; + const cost = val("--cost") !== undefined ? Number(val("--cost")) : null; + try { + const row = U.recordOutcome(root, { task, model: val("--model"), passed, cost }); + if (json) return console.log(JSON.stringify(row, null, 2)); + console.log( + ` recorded ${row.model} ${row.passed ? "pass" : "fail"} for task ${row.task} (.forge/route_outcomes.jsonl)`, + ); + } catch (e) { + console.error(` ${e.message}`); + process.exitCode = 1; + } + return; + } + let rec; + try { + rec = U.routeUniversal(root, task, { + objective: val("--objective"), + provider: val("--provider") ?? "any", + maxDepth: val("--depth") ? Number(val("--depth")) : undefined, + }); + } catch (e) { + console.error(` ${e.message}`); + process.exitCode = 1; + return; + } + if (json) return console.log(JSON.stringify(rec, null, 2)); + if (!rec.ok) { + console.error(` ${rec.reason}`); + process.exitCode = 1; + return; + } + heading( + `${BRAND.brand} route universal — ${rec.objective.kind}${rec.target != null ? ` (target ${rec.target.toFixed(2)})` : ""}\n`, + ); + rec.cascade.forEach((c, i) => { + console.log( + ` ${i === 0 ? "→" : "then, if a check fails →"} ${paint(c.model, "accent")} P(solve alone) ${c.pSolveAlone.toFixed(2)} · ~$${c.expectedAttemptCost.toFixed(3)}/attempt${c.status === "cold" ? " · cold (no outcomes yet)" : ""}`, + ); + }); + console.log( + `\n P(success) ${rec.pSuccess.toFixed(2)} · expected cost $${rec.expectedCost.toFixed(3)} · best single: ${rec.bestSingle.model} ${rec.bestSingle.pSuccess.toFixed(2)} at $${rec.bestSingle.expectedCost.toFixed(3)}`, + ); + console.log( + ` ${rec.candidates} candidate model(s), ${rec.cascadesEvaluated} cascade(s) compared · fit: ${rec.fit.origin}`, + ); + console.log( + ` learn from results: \`${BRAND.cli} route outcome "" --model --pass|--fail --cost \`, then \`${BRAND.cli} route fit\``, + ); +} + HANDLERS.anchor = async (argv) => { const { goalDrift, renderAnchor } = await import("./anchor.js"); const { clearGoal, getGoal, setGoal } = await import("./goal.js"); diff --git a/src/commands.js b/src/commands.js index 217aa76..8bd433f 100644 --- a/src/commands.js +++ b/src/commands.js @@ -92,7 +92,8 @@ export const COMMANDS = { context: "budgeted context assembly + completeness gate — what an edit NEEDS known", preflight: "assumption check — what a task names that the repo doesn't define", config: "provider setup — show / switch / add providers, set default model", - route: "recommend the cheapest capable model for a task (+ gateway config)", + route: + "recommend the cheapest capable model for a task (+ gateway config); `route universal`: any provider's models, lowest expected cost for the success asked for, learned from outcomes", impact: { summary: "hazard-aware blast radius — SCC-aware propagation + data-driven threshold from PageRank centrality and ledger incident history", diff --git a/src/commit_gate.js b/src/commit_gate.js index 6e4eaf0..d8e5c8e 100644 --- a/src/commit_gate.js +++ b/src/commit_gate.js @@ -1,9 +1,16 @@ // forge precommit — the commit-level rung of the gate lattice (turn ⊂ commit ⊂ PR). // The Stop hook gates the TURN and CI's docs check gates the PR; this module runs the // same F1 classifier at the commit boundary so a commit that ships code without its -// doc/state artifact is caught while the fix is still one `git add` away. Same math as -// the paper's Theorem D: each rung is an independent cⱼ layer over the identical -// structural signal, so P(silent miss) falls multiplicatively, not by hope. +// doc/state artifact is caught while the fix is still one `git add` away. +// What this rung does NOT buy (formal synthesis §5.3, corrected 2026-09-21): the rungs +// are not independent. On the same diff the copies of one classifier fire together or +// not at all, so they are nested checks and the residual is (1−p)(1−c_max), not +// (1−p)·∏(1−cⱼ) — repeating a check does not multiply its catch rate. The rung adds +// catches only where it sees what the Stop hook could not: edits made after the turn +// ended, a host or session where the Stop hook never ran (no hook support, +// FORGE_STOPGATE=0, a human committing), or a session whose one Stop block was already +// spent. That is also the portability argument: without hooks, the same check re-binds +// at pre-commit without changing the math. // // Two detectors, both reused — never reimplemented: // (i) COMPLETENESS — classifyPath (gate.js), the same registry-derived total function @@ -11,6 +18,8 @@ // the same commit is a finding. // (ii) SECRETS — hasSecret (secrets.js) over the staged ADDED lines only (context // lines predate the commit), as the built-in fallback when gitleaks is absent. +// A BINARY file gets the format grammars only: its bytes are compressed or random +// by nature, so the entropy leg would measure the encoding, not a credential. // // Modes via FORGE_COMMIT_GATE: "warn" (default — print findings, allow), "block" // (completeness findings refuse the commit), "0"/"off" (kill switch). A secret finding @@ -36,7 +45,11 @@ import { IGNORE_DIRS } from "./util.js"; // So detection is confirmed by the same module's NARROWER redaction rules (format // grammars, PEM, entropy tokens, opaque assigned literals — never a code expression): // one source of truth (secrets.js), calibrated to the verb (mizan). -const lineBlockSecret = (text) => hasSecret(text) && redactSecrets(text) !== text; +/** + * @param {string} text + * @param {{entropy?: boolean}} [opts] + */ +const lineBlockSecret = (text, opts) => hasSecret(text, opts) && redactSecrets(text, opts) !== text; // Exact bytes, no trim — same discipline as gate.js's gitRaw. `gitStrict` THROWS on a git // error or an over-large output (ENOBUFS) — the secret scan turns that into an unscanned @@ -69,7 +82,11 @@ function gitRaw(root, args) { // `--text` defeats a `.gitattributes` `-diff`/`binary` marking (which printed "Binary // files differ" and hid every added line), `--no-textconv` a `diff=` textconv // that rewrites what is shown, `--no-ext-diff` a configured external diff tool. +// `core.quotePath=false` keeps non-ASCII header paths raw, so they match the `-z` paths of +// the binary probe below (a path that still differs only loses the binary relaxation). const DIFF_ARGS = [ + "-c", + "core.quotePath=false", "--literal-pathspecs", "diff", "--cached", @@ -150,6 +167,39 @@ export function scanStagedAdded(root) { return { byFile, unscanned }; } +/** + * Staged paths git itself treats as BINARY (`--numstat` prints `-\t-` for them). A repo can + * mark a text file binary with `.gitattributes`, so the caller also requires content + * evidence (a NUL byte, git's own binary test) before relaxing the scan. Best effort: if + * git fails the set is empty and every file gets the full scan — the strict direction. + * @param {string} root + * @returns {Set} + */ +export function stagedBinaryFiles(root) { + const out = new Set(); + const tokens = gitRaw(root, [ + "--literal-pathspecs", + "diff", + "--cached", + "--numstat", + "-z", + "--no-ext-diff", + "--no-textconv", + ]).split("\0"); + for (let i = 0; i < tokens.length; i++) { + const m = /^(-|\d+)\t(-|\d+)\t(.*)$/s.exec(tokens[i]); + if (!m) continue; + // `-z` rename/copy record: `added\tdeleted\t` NUL old NUL new — the new path is staged. + let path = m[3]; + if (path === "") { + path = tokens[i + 2] ?? ""; + i += 2; + } + if (m[1] === "-" && m[2] === "-" && path) out.add(path); + } + return out; +} + /** * Parse a `--unified=0` diff into the added lines of each file. * @param {string} raw @@ -278,8 +328,13 @@ export function commitGate(root, { env = process.env } = {}) { }; const secretFiles = []; const { byFile, unscanned } = scanStagedAdded(root); + const binary = stagedBinaryFiles(root); for (const [file, lines] of byFile) { - if (lineBlockSecret(lines.join("\n"))) secretFiles.push(file); + const text = lines.join("\n"); + // Binary per git AND per content: a `binary` attribute on a text file (no NUL) + // keeps the entropy leg, so .gitattributes cannot switch it off. + const isBinary = binary.has(file) && text.includes("\0"); + if (lineBlockSecret(text, { entropy: !isBinary })) secretFiles.push(file); } return { ...commitGateDecision({ staged, secretFiles, unscanned, mode }), diff --git a/src/gate.js b/src/gate.js index 4316952..d5bb9e5 100644 --- a/src/gate.js +++ b/src/gate.js @@ -3,9 +3,13 @@ // guarantees a floor: a session that changed code but produced no TEST EVIDENCE (a test // file moved, or a fresh passing `verify` provenance stamp) or moved no doc/state // artifact is blocked ONCE, with the exact repair procedure as the reason. P(silent miss) = -// (1−p)·∏(1−cⱼ) — the gate is the cⱼ≈1 layer for the structural signal "code moved, -// nothing followed". Loop-safe (stop_hook_active + once-per-session marker), fail-open -// on every error path, kill switch FORGE_STOPGATE=0. +// (1−p)·P(no check fires | miss) (formal synthesis Theorem D, corrected 2026-09-21). On +// its proxy, "code moved, nothing followed", the first stop fires exactly (T3). Its catch +// rate on real misses depends on the agent (touching state.md satisfies the docs leg, and +// the block fires once per session) and has not been measured. The same classifier re-run +// at pre-commit or in CI is a NESTED check on the same diff: (1−p)(1−c_max), not a +// product. Loop-safe (stop_hook_active + once-per-session marker), fail-open on every +// error path, kill switch FORGE_STOPGATE=0. // // Classification derives from the SAME registries the atlas is built from (CODE_EXTS/ // DOC_EXTS/config rules) + the shared test-file predicate — no parallel regex lists that @@ -17,7 +21,16 @@ import { execFileSync } from "node:child_process"; import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from "node:fs"; import { extname, join } from "node:path"; import { cusum } from "./anchor.js"; -import { CODE_EXTS, DOC_EXTS, impact, isConfigFile, load as loadAtlas } from "./atlas.js"; +import { + byRelation, + CODE_EXTS, + DOC_EXTS, + fileRelations, + IMPACT_RELATIONS, + impact, + isConfigFile, + load as loadAtlas, +} from "./atlas.js"; import { BRAND } from "./brand.js"; import { readSession, sessionPath } from "./cortex_hook.js"; import { decisionsPath } from "./decide.js"; @@ -277,23 +290,43 @@ export function obligationsFor(classes = {}) { * so it leads with the MISSING leg (test evidence vs docs vs config docs); the old * "handoff alone satisfies the gate" claim survives only on the config-only row, where * that lighter bar is real. Stale-doc candidates come from the CACHED atlas only (a - * hook never builds). + * hook never builds). The same walk names the code files the graph predicts should + * co-change but the session never touched, tagged by relation — the reverse-only walk + * missed the sibling files that were 94.7% of the empirical refutation's misses, so the + * default walks IMPACT_RELATIONS; `relations: ["reverse"]` is the reverse-only option. * @param {string} root * @param {{codeFiles?: string[], driftAlarm?: boolean, - * classes?: {code?: string[], config?: string[], test?: string[]}, row?: string}} [opts] */ + * classes?: {code?: string[], config?: string[], test?: string[], docs?: string[]}, + * row?: string, relations?: readonly string[]}} [opts] */ export function repairReason( root, - { codeFiles = [], driftAlarm = false, classes = {}, row = "code-without-docs" } = {}, + { + codeFiles = [], + driftAlarm = false, + classes = {}, + row = "code-without-docs", + relations = IMPACT_RELATIONS, + } = {}, ) { let likelyDocs = []; + /** @type {string[]} */ + let coChange = []; try { const atlas = loadAtlas(root); if (atlas) { const docs = new Set(); - for (const f of codeFiles.slice(0, 10)) - for (const d of impact(atlas, f, { maxHops: 2 }).impactedFiles) - if (d.endsWith(".md")) docs.add(d); + const reports = codeFiles + .slice(0, 10) + .map((f) => impact(atlas, f, { maxHops: 2, relations })); + for (const r of reports) for (const d of r.impactedFiles) if (d.endsWith(".md")) docs.add(d); likelyDocs = [...docs].slice(0, 5); + const touched = new Set(Object.values(classes).flat()); + for (const f of codeFiles) touched.add(f); + const rels = fileRelations(reports); + coChange = byRelation( + Object.keys(rels).filter((f) => !touched.has(f) && classifyPath(f) === "code"), + rels, + ).map((f) => `${f} (${rels[f]})`); } } catch {} const cited = codeFiles.length ? codeFiles : (classes.config ?? []); @@ -306,6 +339,13 @@ export function repairReason( const handoffStep = (suffix = "") => `\`${BRAND.cli} handoff "" --next ""\` — rewrite the session snapshot the next session resumes from${suffix}.`; const decideStep = `\`${BRAND.cli} decide ""\` if a non-obvious decision was made.`; + const coChangeStep = coChange.length + ? `Co-change candidates the graph predicts but this session never touched — confirm each needs no change: ${coChange + .slice(0, 8) + .join( + ", ", + )}${coChange.length > 8 ? ` (+${coChange.length - 8} more)` : ""}. (reverse = depends on the change · sibling = shares a dependency with it · forward = the change depends on it)` + : ""; let headline; const steps = []; if (row === "code-without-test-evidence") { @@ -329,6 +369,8 @@ export function repairReason( "END-TO-END COMPLETENESS: code changed this session but no doc or state artifact moved with it."; steps.push(docsSyncStep, handoffStep(), decideStep); } + // Second, right after the row's lead step: the files the diff may still owe a change. + if (coChangeStep) steps.splice(1, 0, coChangeStep); if (driftAlarm) steps.push( `Sustained goal drift this session (CUSUM alarm) — re-read the goal: \`${BRAND.cli} anchor\`.`, diff --git a/src/handoff.js b/src/handoff.js index b08af0f..49b6fc3 100644 --- a/src/handoff.js +++ b/src/handoff.js @@ -13,6 +13,23 @@ import { git } from "./util.js"; export const statePath = (root) => join(root, ".forge", "state.md"); +/** + * The ONE size budget for the snapshot, shared by the writer (writeState) and the loader + * (stateBlock), in one unit: UTF-8 bytes of the snapshot body (the provenance line, which + * the loader strips, is not counted). The formal synthesis's T4 correction (2026-09-21) + * found its handoff bounded LINES while its loader injected at most 8 KB, so a valid + * snapshot could be cut at session start; forge had reproduced that mismatch in lines + * (150 written, 80 injected), silently dropping everything past line 80. The writer now + * selects rows until the body fits this budget, so the loader never truncates what the + * writer wrote. 8192 is the synthesis's A5 cap (roughly 2k tokens per session start). + */ +export const STATE_BUDGET_BYTES = 8192; + +const byteLen = (s) => Buffer.byteLength(s, "utf8"); +/** Bytes of `lines` joined by newlines — the measure both sides apply. */ +const bodyBytes = (lines) => + lines.reduce((n, l) => n + byteLen(l), 0) + Math.max(0, lines.length - 1); + /** Branch, dirty files (capped), recent commits — empty-safe outside a git repo. */ export function gatherGitFacts(root, { statusCap = 20 } = {}) { const branch = git(root, ["rev-parse", "--abbrev-ref", "HEAD"]); @@ -61,21 +78,67 @@ export function gatherAssumptions(root, { cap = 5 } = {}) { const arr = (v) => (Array.isArray(v) ? v : v ? [v] : []).map((x) => String(x).trim()).filter(Boolean); -const section = (title, rows, fallback = "- (none)") => [ - `## ${title}`, - ...(rows.length ? rows.map((r) => `- ${r}`) : [fallback]), - "", -]; +const omitted = (n, budget) => `- (+${n} more not kept — over the ${budget}-byte snapshot budget)`; + +/** + * Choose rows in PRIORITY order until the snapshot fits `budget` bytes. `sections` arrive + * in priority order, which is also the display order, so even a loader cut of a + * hand-edited file loses the least important rows first. Every header stays; a section + * whose rows did not all fit ends with an explicit "(+N more not kept)" row, so a drop is + * never silent. Within a section rows keep their given order; a row that does not fit is + * dropped (with the rest of its section) and later, smaller sections may still fit. + * @param {{title: string, rows: string[], fallback?: string}[]} sections + * @param {number} budget + * @returns {string[]} + */ +export function selectSnapshot(sections, budget = STATE_BUDGET_BYTES) { + const head = ["# Session state", ""]; + // Fixed cost first: every header, its blank line, and one reserved line per section — + // its fallback when empty, else the worst-case omission marker. + const reserve = sections.map((sec) => + sec.rows.length ? omitted(sec.rows.length, budget) : sec.fallback || "- (none)", + ); + let used = bodyBytes([ + ...head, + ...sections.flatMap((sec, i) => [`## ${sec.title}`, reserve[i], ""]), + ]); + const kept = sections.map(() => /** @type {string[]} */ ([])); + sections.forEach((sec, i) => { + for (const r of sec.rows) { + const line = `- ${r}`; + const cost = byteLen(line) + 1; // the row plus its newline + if (used + cost > budget) break; + used += cost; + kept[i].push(line); + } + }); + const out = [...head]; + sections.forEach((sec, i) => { + const dropped = sec.rows.length - kept[i].length; + out.push(`## ${sec.title}`, ...kept[i]); + if (!sec.rows.length) out.push(sec.fallback || "- (none)"); + else if (dropped) out.push(omitted(dropped, budget)); + out.push(""); + }); + return out; +} /** * Rewrite the whole snapshot from this session's fields + auto-gathered git facts. * Refuses secrets in the human-supplied fields (same rule as every forge store) and - * truncates to `maxLines` so the session-start injection can never balloon. + * selects rows in the synthesis's A4 priority order (goal, next, decisions, gotchas, + * in-progress, done) until the body fits `budget` — the SAME budget stateBlock injects, + * so the next session reads back exactly what was written. * @param {string} root * @param {{done?:string[]|string, next?:string[]|string, gotchas?:string[]|string, * criteria?:string[]|string, goal?:string, phase?:string}} fields + * @param {{t?: number, budget?: number}} [opts] */ -export function writeState(root, fields = {}, { t = Date.now(), maxLines = 150 } = {}) { +export function writeState( + root, + fields = {}, + { t = Date.now(), budget = STATE_BUDGET_BYTES } = {}, +) { const done = arr(fields.done); const next = arr(fields.next); const gotchas = arr(fields.gotchas); @@ -104,30 +167,33 @@ export function writeState(root, fields = {}, { t = Date.now(), maxLines = 150 } const progress = facts.status.length ? [...facts.status, ...(facts.overflow ? [`(+${facts.overflow} more dirty files)`] : [])] : []; - const lines = [ - "# Session state", - "", - ...section("Goal / Phase", [`${goal}${fields.phase ? ` — phase: ${fields.phase}` : ""}`]), - ...section("Acceptance criteria", criteria), - ...section("Done this session", done), - ...section("Next steps", next), - ...section("Gotchas", gotchas), - ...section("Open assumptions", assumptions), - ...section("In-progress files (git, at handoff)", progress, "- (clean tree)"), - "## Decisions", - `- append-only log: \`.forge/decisions.md\` (\`${BRAND.cli} decide\`)`, - "", - ]; + // A4 priority: goal (with its acceptance criteria), next, decisions, gotchas (with the + // open assumptions — both are "what could bite the next session"), in-progress, done. + const kept = selectSnapshot( + [ + { + title: "Goal / Phase", + rows: [`${goal}${fields.phase ? ` — phase: ${fields.phase}` : ""}`], + }, + { title: "Acceptance criteria", rows: criteria }, + { title: "Next steps", rows: next }, + { + title: "Decisions", + rows: [`append-only log: \`.forge/decisions.md\` (\`${BRAND.cli} decide\`)`], + }, + { title: "Gotchas", rows: gotchas }, + { title: "Open assumptions", rows: assumptions }, + { title: "In-progress files (git, at handoff)", rows: progress, fallback: "- (clean tree)" }, + { title: "Done this session", rows: done }, + ], + budget, + ); const provenance = ``; - const kept = - lines.length + 1 > maxLines - ? [...lines.slice(0, maxLines - 2), "- (truncated to stay bounded)"] - : lines; mkdirSync(join(root, ".forge"), { recursive: true }); writeFileSync(statePath(root), [...kept, provenance, ""].join("\n")); - return { ok: true, path: statePath(root), lines: kept.length + 1 }; + return { ok: true, path: statePath(root), lines: kept.length + 1, bytes: bodyBytes(kept) }; } // Only the EXACT provenance line is stripped — a naive slice at the first "