research: full research programme + pre-registered evaluation refuting two reliability mechanisms

research/README.md

@@ -0,0 +1,84 @@
+# Research
+
+The full research programme behind forgekit: a theory of what a frozen language model
+structurally lacks, an architecture that supplies it, two runnable prototypes, and — most
+importantly — a pre-registered empirical evaluation that **refuted the prototypes' headline
+claims**.
+
+Read in this order. The later work corrects the earlier work, and the corrections are the
+most useful part.
+
+## Start here: what is actually true
+
+| | Claimed (self-built demos) | Measured (real data) |
+|---|---|---|
+| Impact oracle recall | 1.00 | **0.022** — `grep` with no graph beats it ~10× on F1 |
+| Router/gate F1 | 1.00 | **0.37** on 80 real GitHub issues/PRs |
+| Cost saving | +62.1% | **−20.2%** — routing costs *more* than always-premium |
+
+After diagnosing and repairing two defects, with parameters frozen before the held-out
+repositories were touched: recall **0.653**, F1 **0.416**, beating `grep` (0.371) for the
+first time — a real but narrow win.
+
+The general lesson, demonstrated on our own work: **a self-built demonstration can overstate
+field performance by more than an order of magnitude, and careful caveating does not convert
+a demonstration into evidence.**
+
+## The four layers
+
+### 1. [`cognitive-substrate/`](cognitive-substrate/) — the theory
+The originating argument: an LLM is a frozen map `y = f_θ(x)` with three properties —
+statelessness, frozen parameters, bounded context — which structurally deny it five faculties
+(memory, learning, imagination, self-correction, impact-awareness). The remedy is an external
+stateful architecture, not better prompting.
+
+- `cognitive_substrate_whitepaper.pdf` — the *Theory → Evidence → Build-Map* edition (48pp)
+- `EXECUTIVE_SUMMARY.md` — one-page entry point, **carries a status banner: its prototype numbers are refuted**
+- `literature/` — the gap map and 32 graded references behind each faculty claim
+- `evidence/` — twelve load-bearing industry statistics independently re-grounded and graded
+  `confirmed` / `vendor-reported` / `unverifiable`, plus an ecosystem map of what the 2026
+  Claude-Code stack already solves. Three widely-repeated statistics were caught as
+  misattributed and dropped.
+- `quranic-lens/` — the fourteen-mapping ethical-epistemic reading used as a *design lens*:
+  it names which safeguards are obligatory rather than optional. It is framing, never
+  technical authority; no verse is offered as proof of an engineering claim.
+- `sources/` — the primary documents the evidence layer was graded against
+- `figures/` — the architecture schematics and prototype evaluations
+
+### 2. [`formal-synthesis/`](formal-synthesis/) — the mathematics
+Unifies the substrate theory, the end-to-end reliability framework (F1/F2, Δ*, I1–I4, A1–A7,
+T1–T6), and the forgekit implementation, proving they are one object in three vocabularies.
+Central result is a two-layer duality: reliability = a probabilistic instruction layer × a
+deterministic interception layer, neither alone sufficient.
+
+**Priority note:** prior-art review found this composition law is standard protection-layer
+algebra, and two concurrent preprints derive a strictly more general Bayesian form weeks
+earlier. Priority is conceded in the paper. What survives is that both are simulation-only.
+
+### 3. [`empirical-refutation/`](empirical-refutation/) — the measurement
+The pre-registered evaluation that overturned the claims above, the diagnosis of *why*, and
+the repair. Includes a replication package with the frozen pre-registration, mined ground
+truth, held-out task set, every result with bootstrap confidence intervals, both prototype
+versions, and **our own adversarial review listing four numbers we had stated incorrectly
+before correction**.
+
+Also corrects a theoretical claim: perfect recall was inferred from a completeness theorem,
+but such a theorem guarantees completeness only *relative to the relation* the closure runs
+over — it says nothing about whether that relation contains the edges that matter.
+
+### 4. [`python-prototypes/`](python-prototypes/) — the code
+`impact_oracle/` and `router_gate/`, runnable with their own test suites. The **repaired**
+oracle ships inside the refutation's replication package rather than replacing the version
+here, so swapping it in stays a deliberate decision.
+
+## How this programme tries to stay honest
+
+- Protocols and parameter freezes are declared *before* the data is seen, and amendments are
+  append-only.
+- Every load-bearing statistic is graded by what can actually be traced to a primary source.
+- Negative results are reported as findings, not tuned away.
+- Reviews of our own work ship alongside it, including the parts that found us wrong.
+
+Where this falls short is stated too: the pre-registration and parameter freezes were
+self-administered with no external timestamping authority, so a reader can verify internal
+consistency and the amendment trail but must take the ordering on trust.

research/cognitive-substrate/EXECUTIVE_SUMMARY.md

@@ -0,0 +1,102 @@
+> ## ⚠ Status: headline claims in this document are REFUTED
+>
+> This summary describes the *Theory → Evidence → Build-Map* edition (v2), written before any
+> real-repository evaluation existed. A later pre-registered evaluation — see
+> [`../empirical-refutation/`](../empirical-refutation/) — overturned both prototype claims reported below:
+>
+> | Claim below | Measured on real data |
+> |---|---|
+> | Impact oracle recall **1.00** | **0.022** (9 OSS repos, 801 files); `grep` beats it ~10× on F1 |
+> | Router/gate F1 **1.00**, cost saving **+62.1%** | F1 **0.37**; cost saving **−20.2%** (routing costs *more* than always-premium) |
+>
+> The theory sections remain the programme's working framework. The *numbers* here do not. A repair
+> recovered a narrow win (recall 0.653, F1 0.416, beating grep's 0.371), documented in the refutation
+> paper. Read this document for the architecture; read the refutation for what is actually true.
+
+---
+
+# A Cognitive Substrate for Coding Agents — Deliverable Package
+### Theory → Evidence → Build-Map edition (v2)
+
+**One-line thesis:** The faculties a coding agent lacks — memory, learning, imagination, self-correction, impact-awareness — are not gaps in the model's *knowledge* but structural consequences of what a frozen transformer *is* (a stateless map `y = f_θ(x)`, fixed weights, bounded window). They cannot be prompted or tooled away; they can only be supplied by **re-wrapping the input→process→output loop** into a closed, stateful cycle around the frozen model.
+
+**What v2 adds.** The first edition argued the five faculties from first principles and prototyped the one that is buildable today. This edition (1) **grounds the argument in the field's own evidence** — twelve load-bearing pain-point statistics independently re-grounded from primary sources and graded *confirmed / vendor-reported / unverifiable*; (2) adds **six metacognitive mechanisms** the frozen loop also lacks (routing, assumption gate, decomposition, goal-anchoring, anti-over-engineering, inline verification); (3) **maps all eleven capabilities against the real 2026 Claude-Code stack**, marking each solved / partial / residual-gap so we say clearly *what not to build*; and (4) ships a **second runnable prototype** — a complexity-aware router + assumption gate, evaluated live on real models.
+
+> **Governing discipline (the user's, adopted throughout):** *AI output is mathematically-calculated probability — non-deterministic, and never blindly trusted.* Every claim in this package is graded by how well it is sourced; every prototype decision is a transparent, attributable rule rather than another opaque model call; and trust is always earned by an **external** check, never asserted by the model.
+
+---
+
+## What's in this package
+
+### 1. The white paper (core deliverable) — 48 pp
+- **`cognitive_substrate_whitepaper.pdf`** / **`cognitive_substrate_whitepaper.html`** — the full study, 13 sections + 3 appendices, 7 figures.
+  - **§1–3** the root cause and the five faculties (from v1): *why* each faculty is structurally absent (P1 statelessness, P2 frozen weights, P3 bounded context), each grounded in the real literature.
+  - **§4 Evidence** *(new)* — the twelve statistics, re-grounded. 5 confirmed, 5 vendor-reported, 2 unverifiable.
+  - **§5** the Qur'anic epistemic lens — design framing/ethics, never technical authority.
+  - **§6 Six mechanisms** *(new)* — M1 routing, M2 assumption gate, M3 decomposition, M4 goal-anchoring, M5 anti-over-engineering, M6 inline verification — each formalized, with ecosystem status and a Qur'anic anchor.
+  - **§7** the cognitive substrate, now with the six-mechanism metacognitive control layer (Figure 3).
+  - **§8 Prototype I** the impact oracle (from v1). **§9 Prototype II** *(new)* the router + gate. **§10 Build-map** *(new)* the ranked opportunity list.
+  - **§11** new-vs-reinvented. **§12** limitations. **§13** conclusion.
+
+### 2. Prototype I — Codebase World-Model + Impact Oracle
+- **`impact_oracle_src.zip`** — parses a codebase (AST) into a **persistent dependency graph**, predicts the **blast radius** of a proposed edit via reverse-dependency traversal with confidence decay. `python demo.py` runs end-to-end; `pytest` → **36 tests pass** with zero setup. Builds opportunity #3.
+
+### 3. Prototype II — Complexity-aware router + Assumption gate  *(new)*
+- **`router_gate_src.zip`** — the two mechanisms at the top of the build-map, composed as `gate → route → execute → verify → escalate`. Both are **transparent additive rubrics**, not opaque LLM calls; escalation is driven by an external check. `python demo.py`, `pytest` → **19 tests pass**, `python evaluate.py --live` reproduces the live numbers.
+- **`eval_results.json`** — the live evaluation record (real measured tokens).
+
+### 4. Evidence & ecosystem maps  *(new)*
+- **`evidence_map.json`** / **`evidence_map.md`** — every load-bearing statistic, its primary source, and its status.
+- **`ecosystem_map.json`** / **`ecosystem_map.md`** — every faculty & mechanism vs. the real stack, with residual gap and proposed contribution.
+
+### 5. Figures & schematics
+- **`schematic_loop.png`** (Fig 1) · **`schematic_system.png`** (Fig 2) · **`schematic_extended.png`** (Fig 3, *new* — the six-mechanism control layer) · **`impact_graph.png`** (Fig 4) · **`eval_precision_recall.png`** (Fig 5) · **`schematic_router_loop.png`** (Fig 6, *new*) · **`router_eval.png`** (Fig 7, *new*).
+
+### 6. Supporting artifacts (from v1)
+- **`gap_map.json`/`.md`**, **`references.json`** (32 sources), **`quran_lens.json`/`.md`** (14-row concept→faculty→design-principle mapping).
+
+---
+
+## The honest headline results
+
+### Prototype I — Impact Oracle (against mutation-derived ground truth, 5 real edits)
+| Method | Precision | Recall | F1 |
+|---|---|---|---|
+| **Graph Oracle** (ours) | 0.63 | **1.00** | 0.75 |
+| Grep baseline (what agents do today) | 0.73 | 0.94 | **0.79** |
+| Edited-file-only | 1.00 | 0.53 | 0.65 |
+
+The oracle does **not** dominate F1 — grep edges it at the default threshold, and we say so. What the oracle uniquely provides is **guaranteed recall**: for "show me everything my edit could break," a silent miss costs far more than an extra file to check, and only the structural oracle drives false negatives to zero (precision tunable, best F1 = 0.79 at threshold 0.4).
+
+### Prototype II — Router + Gate (live, on real models: haiku / sonnet / opus)
+| Metric | Result |
+|---|---|
+| Gate accuracy (should-ask) | 30/30 · precision 1.00 · recall 1.00 |
+| Routing accuracy (well-specified tasks) | 21/21 exact tier |
+| **Real cost saved vs always-premium** | **62.1%** (same measured tokens) |
+| Execution-verified sub-experiment | 3/3 routed-down outputs passed real test cases |
+
+**Honest caveat (both prototypes):** these are **demonstrations, not benchmarks**. The router's 30-task set is hand-labeled and the rubric thresholds were tuned against it, so perfect separation shows the rubric *can* distinguish these cases — not field accuracy. The oracle's evaluation is 5 mutations + 2 stdlib scale checks. We apply the "retired SWE-bench Verified" caution (§4, confirmed) to our own numbers.
+
+## What the evidence re-grounding caught
+
+The independent re-grounding **changed our claims** — three widely-repeated numbers did not survive and are *not* used as fact in this paper:
+- **"2.74× more vulnerabilities"** is not traceable to Veracode's own report (only their 45% OWASP figure is); likely conflated with a separate study.
+- **"17% lower comprehension / 400K sessions"** merges two different studies — the session study contains no comprehension finding.
+- **GitClear 4× vs 8×** internal inconsistency and **JetBrains 77%** could not be located in primary form.
+
+That a re-grounding pass corrected the paper is the point, not an embarrassment: it is the same discipline the architecture makes structural — *a stored fact is provisional until an external check confirms it.*
+
+## The build-opportunity map (what to build, what to skip)
+
+**Already solved — do not rebuild:** M1 routing (model tiering + gateways like LiteLLM/OpenRouter) and M3 decomposition (subagents, Agent-Teams). The router prototype's honest contribution is only the *transparency layer*, and we say so.
+
+**The genuine whitespace, ranked:** (1) **assumption/uncertainty gate** — the project's named root failure and the field's named gap; nothing supplies calibrated known-unknowns. (2) **validity-anchored memory** — backends store notes, none tracks invalidation-by-correction. (3) **mandatory pre-action impact gate** — indexers retrieve, none is a deterministic blast-radius check. (4) outcome-validated learning. (5) doom-loop / root-cause correction. (6) scope-minimality. This paper prototypes #1 and #3 — the two where a single session can produce checkable ground truth.
+
+## What is genuinely new vs. reinvented
+
+Most components are borrowed (external memory, fast/slow learning, code graphs, model tiering — all exist). The contribution is **the composition and the framing**: the closed-loop shape; **validity-anchored memory** (prune by whether a past prediction was confirmed by an *external* oracle, not by the model's own judgment); wiring exact impact analysis into a **mandatory pre-action gate**; a **transparent** router/gate that explains every decision; and deriving *which* safeguards are non-negotiable from a coherent epistemology. That turns scattered literatures and named-but-unsolved gaps into one buildable architecture aimed squarely at coding agents.
+
+## Scope & limitations (stated honestly)
+
+Two faculties/mechanisms are prototyped, not eleven. The impact oracle's static analysis is single-language (Python) and conservative on dynamic dispatch. The router/gate rubrics are keyword heuristics tuned on a small hand-labeled set. Memory validity, outcome learning, and doom-loop diagnosis remain *specified but unbuilt* — the harder research gaps, marked as such rather than gestured at with a demo. The lens is framing: reject it and you lose the organizing vocabulary but none of the technical content.