From 9ebe256096297f13696e321c28f674f435c8adcb Mon Sep 17 00:00:00 2001
From: Juber Shaikh <40266375+CodeWithJuber@users.noreply.github.com>
Date: Sun, 20 Sep 2026 13:55:07 +0200
Subject: [PATCH 1/3] research(refutation): pre-registered evaluation refutes
two reliability mechanisms
Adds research/empirical-refutation/: a pre-registered empirical evaluation of the
impact oracle and the complexity-router/assumption-gate against real repositories
and real issues, which REFUTES both of their previously reported headline claims.
Findings:
- Impact oracle recall 1.00 (self-built demo) -> 0.022 (9 OSS repos, 801 files).
A grep baseline with no dependency graph beats it ~10x on F1.
- Router/gate F1 1.00 -> 0.37 on 80 real issues/PRs, thresholds frozen.
- Cost saving does not shrink but inverts: counting the pipeline's actual
escalation spend, routing costs 20.2% MORE than always-premium.
Diagnosis and repair:
- 96.8% of missed impact was reachable in the graph the tool already builds:
94.7% sibling pairs unreachable by reverse-only traversal, plus a construction
defect affecting src-layout packages. Only 3.2% was a true proxy ceiling.
- Repairing both, with parameters frozen before three held-out repos were touched,
lifts recall 0.022 -> 0.653 and F1 0.042 -> 0.416, beating grep (0.371) for the
first time. The margin is narrow and precision generalised worse than recall.
Also corrects a theoretical claim: perfect recall was inferred from Theorem T5,
but T5 guarantees completeness only relative to the relation the closure runs over.
It says nothing about whether that relation contains the edges that matter.
Includes the frozen pre-registration, mined ground truth, held-out task set with
two labelling passes, all results with bootstrap CIs, both prototype versions, a
replication package, and our own adversarial review listing four numbers we had
stated incorrectly before correction.
---
research/empirical-refutation/README.md | 96 +++
.../extended_preprint.html | 783 ++++++++++++++++++
.../extended_preprint.pdf | Bin 0 -> 791707 bytes
.../figures/fig_cochange_results.png | Bin 0 -> 126192 bytes
.../figures/fig_heldout_calibration.png | Bin 0 -> 165483 bytes
.../figures/fig_heldout_comparison.png | Bin 0 -> 109084 bytes
.../figures/fig_precision_recall_curve.png | Bin 0 -> 219697 bytes
.../figures/fig_repair_beforeafter.png | Bin 0 -> 161702 bytes
.../figures/fig_threshold_sweep.png | Bin 0 -> 164032 bytes
research/empirical-refutation/paper.pdf | Bin 0 -> 829517 bytes
research/empirical-refutation/paper/main.tex | 552 ++++++++++++
research/empirical-refutation/paper/refs.bib | 395 +++++++++
.../replication_package.tar.gz | Bin 0 -> 1929355 bytes
13 files changed, 1826 insertions(+)
create mode 100644 research/empirical-refutation/README.md
create mode 100644 research/empirical-refutation/extended_preprint.html
create mode 100644 research/empirical-refutation/extended_preprint.pdf
create mode 100644 research/empirical-refutation/figures/fig_cochange_results.png
create mode 100644 research/empirical-refutation/figures/fig_heldout_calibration.png
create mode 100644 research/empirical-refutation/figures/fig_heldout_comparison.png
create mode 100644 research/empirical-refutation/figures/fig_precision_recall_curve.png
create mode 100644 research/empirical-refutation/figures/fig_repair_beforeafter.png
create mode 100644 research/empirical-refutation/figures/fig_threshold_sweep.png
create mode 100644 research/empirical-refutation/paper.pdf
create mode 100644 research/empirical-refutation/paper/main.tex
create mode 100644 research/empirical-refutation/paper/refs.bib
create mode 100644 research/empirical-refutation/replication_package.tar.gz
diff --git a/research/empirical-refutation/README.md b/research/empirical-refutation/README.md
new file mode 100644
index 0000000..afe3e8b
--- /dev/null
+++ b/research/empirical-refutation/README.md
@@ -0,0 +1,96 @@
+# Replication package
+
+*Static Impact Analysis Does Not Transfer: A Pre-Registered Refutation of Two LLM-Agent Reliability Mechanisms*
+
+This package contains everything needed to check every number in the paper. It is organised so that
+a reviewer can start from the frozen protocol and work forward, in the order the work was actually
+done.
+
+**Read `protocol/PRE_REGISTRATION.md` first.** It was written and frozen before any repository was
+cloned, and it fixes every threshold, filter, and selection rule used downstream. Its own stated
+rule is the one thing that makes the rest of this package meaningful: *if a downstream number looks
+bad, the fix is to report it, not to edit this file.* Amendments were permitted only as dated,
+append-only addenda; three were filed, all documenting the corpus-selection funnel.
+
+## What is here
+
+### `protocol/` — the frozen contract
+| File | What it is |
+|---|---|
+| `PRE_REGISTRATION.md` | Nine corpus eligibility criteria, commit filters, threshold grid, bootstrap seeds. Frozen before data collection. |
+| `corpus_manifest.json` | The nine repositories with exact clone SHAs, the eligibility decisions, and every substitution logged. |
+| `SPLIT_DECLARATION.json` | The tuning/held-out repository split for the repair, declared before any repair code was written. |
+| `FROZEN_PARAMETERS.json` | The eight repair parameters, fixed on tuning repositories only, before the held-out set was touched. |
+
+### `data/` — the ground truth
+| File | What it is |
+|---|---|
+| `cochange_groundtruth.parquet` | 801 labelled files across 9 repositories. For each file, the other Python files that co-changed with it in the same non-merge commit, after the pre-registered filters. |
+| `heldout_taskset.json` | 80 tasks drawn from real GitHub issues and PRs, with full provenance (repo, number, URL, verbatim text), gold labels, the labelling protocol, and a second independent labelling pass. |
+
+### `results/` — every number in the paper
+| File | What it is |
+|---|---|
+| `cochange_results.json` | Per-repo and pooled P/R/F1 with bootstrap CIs, the full threshold sweep, the per-file recall distribution. |
+| `cochange_failure_modes.json` | All 18,221 false negatives classified structurally, with the ground-truth ceiling. |
+| `cochange_report.md` | The written evaluation, including threats to validity. |
+| `heldout_results.json` | Routing and gate metrics with CIs, inter-rater agreement, calibration bins, and the full cost accounting. |
+| `repair_results.json` | The (a)–(g) repair sequence: as-shipped reproduction, each fix, tuning and held-out results. |
+| `repair_report.md` | The written before/after. |
+| `novelty_assessment.json` | Five claimed contributions graded against closest prior work, including the scoop. |
+| `related_work.md`, `related_references.json` | The full survey and 52 graded references. |
+| `internal_review_findings.json`, `internal_review_report.md` | Our own adversarial review of this paper, and what it found wrong. Included deliberately: it lists four numbers we had stated incorrectly before correction. |
+
+### `prototypes/` — the code under test
+| File | What it is |
+|---|---|
+| `impact_oracle_v1_as_shipped.zip` | The version whose claims the paper refutes. 36 tests. |
+| `impact_oracle_v2_src.zip` | The repaired version. 49 tests, including the stdlib-collision safety case. |
+| `router_gate_src.zip` | The router and assumption gate, thresholds exactly as evaluated. 19 tests. |
+
+Each package runs with `python -m pytest` from its own root (a `conftest.py` handles the path).
+The oracle needs `networkx`; the router needs only `pytest`.
+
+### `figures/`, `paper/`
+The six figures at full resolution, and the paper with its LaTeX source and bibliography.
+
+## Verifying the headline claims
+
+**The refutation.** `results/cochange_results.json` → `pooled_metrics.oracle_by_threshold["0.02"]`
+gives precision 0.398, recall 0.022, F1 0.042 from tp/fp/fn = 409/618/18221. The grep baseline in the
+same file gives F1 0.437. Every F1 in the paper is computed from raw counts and rounded once.
+
+**The ceiling.** `results/cochange_failure_modes.json` → the three failure categories
+(94.68% sibling, 2.13% forward-only, 3.19% no static path) sum over the 18,221 false negatives.
+The 3.19% figure is what bounds achievable recall at 96.88%.
+
+**The repair.** `results/repair_results.json` → `d_defect1and2_heldout_HEADLINE`. The paper headlines
+`metrics_at_canonical_0.02` (F1 0.416), not the higher `metrics_at_best_threshold` (0.428), because
+the latter's threshold was selected on the tuning repositories.
+
+**The held-out collapse.** `results/heldout_results.json` → `tuned_vs_heldout_comparison`.
+Note `cost_analysis.n_execution_verified = 0`: no held-out task admitted execution-based
+verification, so correctness used a weaker model-based criterion.
+
+**The cost inversion.** `results/heldout_results.json` → `cost_analysis` carries four figures along two
+orthogonal axes, and the paper reports all four rather than the most favourable one. Framing:
+`first_attempt_framing` counts only the initially-routed attempt; `escalation_inclusive_framing` counts
+everything the pipeline spent retrying up the tier ladder. Gating: `raw_saving_pct` credits every dollar
+saved, `correctness_gated_saving_pct` credits only dollars saved on output that verifies. The tuned
+demonstration's 62.1% corresponds to first-attempt/ungated (59.5% held out). The honest total-spend,
+ungated figure is **−20.2%**: the router costs more than always using the premium tier. The source's own
+`honesty_note` states this.
+
+## What this package cannot establish
+
+The pre-registration, the split declaration, and the parameter freeze were all self-administered
+within one continuous working session. There is no external timestamping authority. A reader can
+verify internal consistency and the append-only amendment trail, but must take the ordering on
+trust. We regard this as the central weakness of a self-evaluation and state it in the paper rather
+than resting on the protocol's authority.
+
+Co-change is a proxy for semantic impact and errs in both directions: files co-change for reasons no
+static analysis can predict, and an over-warning may be a correct dependency that has not yet
+co-changed. The 96.9% ceiling is measured on the graph the as-shipped oracle builds, and reachability
+in a dense graph is a weak property — it bounds what any static method could attain, and is not
+evidence that a reachable pair is causally related.
diff --git a/research/empirical-refutation/extended_preprint.html b/research/empirical-refutation/extended_preprint.html
new file mode 100644
index 0000000..904831c
--- /dev/null
+++ b/research/empirical-refutation/extended_preprint.html
@@ -0,0 +1,783 @@
+
+
+
+
+
+A Formal Theory of the Cognitive Substrate for Coding Agents
+
+
+
+
+
+
A Formal Theory of the Cognitive Substrate for Coding Agents
+
Extended edition, with a pre-registered empirical refutation. Unifies the substrate faculties, the end-to-end reliability framework, and the forgekit implementation — with a two-layer duality theorem, a unified algorithm set, a Qur’anic epistemology carried in full, and a measurement that overturned two of this work’s own headline claims.
+
A synthesis edition. Consolidates three independently-developed bodies of work: the Cognitive Substrate for Coding Agents (faculties and mechanisms M1–M6, with two runnable prototypes), the End-to-End Agent Reliability Framework (failure modes F1/F2, invariants I1–I4, algorithms A1–A7, theorems T1–T6), and the forgekit / claude-e2e-kit implementation (deterministic hooks, skills, and committed-file memory). The fourteen-mapping Qur'anic lens is carried throughout as the governing epistemology of obligation.
+
+
Status of this edition
+
This is the extended companion to a venue submission reporting a pre-registered empirical evaluation of
+the two prototypes described here. That evaluation refuted both of their headline claims:
+the impact oracle's perfect recall collapsed from 1.00 to 0.022 on real repositories, and the router/gate
+pair's perfect separation fell to F1 = 0.37 with its cost saving inverting from +62.1% to −20.2% once the pipeline's escalation retries are counted. Section 10 reports the refutation, the diagnosis, and a repair that recovers a
+narrow win over the baseline, and states what the failure costs the formalism — specifically, that
+Theorem T5's completeness guarantee transfers nothing to practice until the underlying relation is
+shown adequate. Theory sections are otherwise unchanged; where they make empirical claims, those claims are
+now the corrected ones.
+
+
+
+
Abstract
+
A large language model used for coding is a fixed probabilistic map, y = fθ(x): stateless, frozen, and bounded in context. Three research efforts converged, independently, on the same conclusion — that the remedy is not a better prompt or a bigger model but an external, stateful architecture wrapped around the frozen core. This paper proves they are describing one object. We show that the substrate's impact-awareness faculty and the framework's change-closure fixpointΔ* are the same mathematics; that the assumption gate and the amnesia equationassumption ≈ argmax P(convention | training) are the same phenomenon; and that both reduce to a single two-layer duality: a probabilistic instruction layer that raises the probability p<1 of correct behaviour, and a deterministic interception layer that guarantees a floor. The central theorem states that neither layer alone can make an agent reliable — a direct formalization of the discipline never trust the output of a probabilistic engine; earn trust with an external check. We give definitions, the duality theorem with proof, a unified seven-algorithm task loop, the probabilistic failure model P(≥1 miss)=1−pn, and carry through the six correctness theorems of the reliability framework. Two prototypes — an impact oracle and a complexity-router/assumption-gate — instantiate the deterministic layer and are evaluated honestly. The forgekit / claude-e2e-kit codebase is the deployed binding. The Qur'anic lens supplies the vocabulary of epistemic obligation (tabayyun, amāna, lā taqfu) that names why each safeguard is mandatory rather than optional.
1 The convergence — three roads to one architecture
+
+
Three efforts set out from different starting points and arrived at the same building.
+
+
The first began with a question about human cognition: when a developer opens a file, they carry memory of what already exists, imagine what an edit will break, and verify their reasoning as they type. A frozen language model does none of this. The Cognitive Substrate workS named five faculties the model structurally lacks — memory, learning, imagination, self-correction, and impact-awareness — and argued the remedy is an external architecture that supplies them, together with six operating mechanisms (M1–M6: complexity-routing, an assumption gate, decomposition, goal-anchoring, scope-minimality, and inline verification).
+
+
The second began with two concrete failures observed in production coding agents. F1, partial work: the agent changes code but not the artifacts that depend on it — docs, tests, changelog, examples. F2, session amnesia: a later session lacks the project's goals and conventions, so it fills the gaps with assumptions and the developer burns iterations re-explaining. The End-to-End Reliability FrameworkE gave these a formal model: a typed artifact graph, a change-closure fixpoint, four invariants, seven algorithms, and six correctness theorems.
+
+
The third is a deployed codebase. forgekitK (and its Claude-specific precursor claude-e2e-kit) implements the same discipline as committed files, deterministic lifecycle hooks, and auto-invoked skills — one configuration that binds the architecture onto Claude Code, Codex, Cursor, Gemini, and Aider alike.
+
+
+
The claim of this paper
+
These are not three similar ideas. They are one architecture described in three vocabularies. The impact-awareness faculty is the change-closure fixpoint. The assumption gate is the amnesia equation. The substrate's external structure is a two-layer duality — and that duality, which the reliability framework states as a design law, is the theorem the whole thing turns on. What each road saw partially, the union sees whole.
+
+
+
The synthesis also inherits a governing discipline, stated plainly by the practitioner who commissioned this work: AI output is a mathematically calculated probability; it must never be trusted blindly; for the same prompt it can give a different answer, so use only the capability it is genuinely best at, and earn trust with an external check. We will see that this sentence is not a slogan but the informal statement of the central theorem — the quantity (1−p)>0 that forces a deterministic layer to exist.
+
+
2 The object of study — the frozen map and its five lacks
+
+
Fix the model. Let the coding agent's core be a function
+
(1)y = fθ(x), θ fixed, x the bounded context window, y the sampled output.
+
+
Three properties of this map generate every problem the architecture must solve. To avoid a notation collision with the reliability framework's primitives (§3), we label these model propertiesP1–P3:
+
+
+
P1 — statelessness.fθ has no memory across calls; each invocation sees only the current x. Nothing the agent learned yesterday is present today unless something outside the model re-supplies it.
+
P2 — frozen parameters.θ does not change from use. The agent cannot learn from an outcome by updating weights; any learning must be external.
+
P3 — bounded, undifferentiated context.x is finite and flat: a long story and a long program are the same kind of object to it, with no privileged channel for goals versus detail. This is the root of goal-drift and of context saturation.
+
+
+
The five faculties are not wishes; each is the direct consequence of one or more of these properties, and each has an external remedy:
+
+
+
Faculty the model lacks
Forced by
External remedy (this architecture)
+
Persistent memory
P1
A committed store re-injected each session (§6, A4/A5)
A typed artifact graph the agent queries before acting (§3.2)
+
Self-correction
P3
An external verifier and a deterministic completion gate (§4, A6)
+
Impact-awareness
P1, P3
The change-closure Δ* computed on the graph (§3.2, A1)
+
+
+
The critical word is external. Because θ is frozen (P2) and context is bounded (P3), none of these can be fixed by prompting harder or by fine-tuning alone. The architecture must live around the model, hold state outside it, and enforce behaviour the model cannot be relied upon to produce on its own. The rest of this paper makes "cannot be relied upon" precise and shows what "enforce" must therefore mean.
+
+
+
3 Definitions
+
+
3.1 The stateful substrate operator
+
+
The architecture turns the stateless map (1) into a stateful operator by threading an external memory M through it.
+
+
Definition 1 (Cognitive substrate)
+
A cognitive substrate over a frozen model fθ is an operator
+
(2)(yt, Mt+1) = F(xt, Mt; fθ)
+
where Mt is a persistent store that survives between calls, xt is the request at step t, and yt is the output. The store carries what P1–P3 deny the model: prior state, learned priors, the world-model, and the reflection log. F is required to readMt into the model's context before sampling and to write backMt+1 after — the closed loop the bare model (1) does not have.
+
+
+
Three environmental capabilities are needed to realize M and the read/write loop. The reliability framework calls these its primitives. It labels them P1/P2/P3, which collides with the model properties of §2; we therefore rename them Π1, Π2, Π3 and use that notation for the remainder of the paper.
+
+
+
Primitive
Definition
forgekit binding
Generic binding
+
Π1 — persistent store
Files the agent can read/write that survive sessions and travel with the project
git repo: CLAUDE.md, docs/*.md, .claude/**
any VCS; any repo-reading agent
+
Π2 — lifecycle interception
Deterministic code executed at fixed points of the agent loop (start, end-of-turn)
hooks: SessionStart, Stop, UserPromptSubmit
pre-commit hooks; CI jobs; IDE tasks
+
Π3 — instruction channel
Standing instructions loaded into the model's context
CLAUDE.md, .claude/rules/, skills
AGENTS.md, .cursorrules, system prompts
+
+
+
+
The design law, stated early because everything depends on it
+
Π3 is probabilistic; Π2 is deterministic. Instructions (Π3) raise the probability that the model behaves correctly; interception (Π2) executes regardless of what the model decides. A reliable substrate needs both, and §4 proves it cannot be built from either alone.
+
+
+
3.2 The repository as a typed artifact graph (the world-model)
+
+
The imagination/world-model faculty is realized concretely as a graph over the project's artifacts — the generalization that lets "impact on code" become "impact on everything that describes or depends on the code".
+
+
Definition 2 (Typed artifact graph)
+
Let the project be a finite set of artifacts A = {a1, …, an} with a type function τ : A → {code, test, doc, config, diagram} and a dependency relation R ⊆ A × A, where (a,b) ∈ R means "a describes, verifies, exercises, or references b". R = Rdeclared ∪ Rdiscovered: declared edges come from a curated documentation map (high precision, small); discovered edges are found mechanically — a mentions an identifier defined in b — by text search.
+
+
+
The substrate's Prototype I builds this graph from source: an AST parser extracts the code nodes and their edges, so Rdiscovered over {code, test} is computed exactly rather than by grep. The reliability framework's contribution is to widen τ beyond code, making documentation a first-class dependent so that a code change can be seen to obligate a doc change.
+
+
Definition 3 (Dependents operator and change closure)
+
For a set X ⊆ A, the dependents operator is
+
(3)N(X) = { a ∈ A : ∃ b ∈ X, (a,b) ∈ R }
+
— everything that describes, verifies, or references anything in X. A task seeds a change set Δ0 ⊆ A (the files the request names or obviously touches). The required change closure is the least fixpoint
+
(4)Δk+1 = Δk ∪ N(Δk), Δ* = Δk where Δk+1 = Δk.
+
Since A is finite and the sequence is monotone (Δ0 ⊆ Δ1 ⊆ … ⊆ A), the fixpoint exists and is reached in at most |A| steps (Kleene's theorem on a finite lattice). In practice depth 2–3 suffices.
+
+
+
+
Anchor identity #1: the impact oracle is Δ*
+
The substrate's Impact Oracle computes a file's blast radius by reverse reachability on the dependency graph — which is exactly the closure (4). The oracle adds a real-valued confidence that decays with graph distance, where the framework's N(·) is boolean; thresholding the oracle's confidence recovers N. They are the same computation. This is why the prototype achieves perfect recall on impacted files (§10): reverse reachability, run to fixpoint, cannot miss a reachable dependent.
+
+
+
3.3 The faculties as operators on the store
+
+
Each faculty is a well-typed operation on M, and each will be realized by one of the algorithms A1–A7 (§6):
+
+
Definition 4 (Faculty operators)
+
+
retrieve: (x, M) → c — select the context c ⊆ M relevant to x, favouring entries whose validity has been externally confirmed (validity-anchored memory). (memory; A5)
simulate: (edit, M) → predicted-effects — consequence estimation over the graph before acting. (imagination; A1)
+
verify: (y, criteria) → {pass, fail, doubts} — an external check, never the generator's self-report. (self-correction; A3, A6)
+
route: x → tier and : fact → store-home — transparent classification of effort and of knowledge placement. (M1; A7)
+
write-back: (M, outcome) → M′ — the bounded-compression checkpoint that makes learning survive P1/P2. (memory + learning; A4)
+
+
+
+
With the object (Def. 1), the world-model (Defs. 2–3), and the faculties (Def. 4) in hand, we can state the result the architecture rests on.
+
+
+
4 The central result — the two-layer duality theorem
+
+
Everything so far has assumed that "the architecture must enforce behaviour the model cannot be relied upon to produce." We now make "cannot be relied upon" precise and derive what "enforce" must mean.
+
+
Definition 5 (The two layers)
+
A substrate's behaviour-shaping is partitioned into two layers over the frozen model:
+
+
The probabilistic layer (Π3): standing instructions loaded into context — CLAUDE.md, rules, skills, protocol cards. Let p = P(the agent performs the required behaviour on a task — e.g. the full closure Δ* — under the instruction layer alone).
+
The deterministic layer (Π2): code executed at fixed lifecycle points independent of the model's choices — hooks that inject state, or that block a turn. A deterministic check j catches a target miss with probability cj, and for a decidable structural signal cj → 1.
+
+
+
+
Theorem D (Two-layer duality — neither layer alone suffices)
+
Let a task require a behaviour whose omission is a silent miss. Under the instruction layer alone the per-task silent-miss probability is 1−p; under a deterministic layer of k checks the residual silent-miss probability is
+
(5)P(silent miss) = (1−p) · ∏j=1..k (1−cj).
+
Then, for any model whose instruction-following is imperfect (p<1) and any deterministic layer that is not omniscient (cj<1 for every j):
+
+
The probabilistic layer alone cannot reach reliability. With k=0, P(silent miss)=1−p>0, and over n tasks P(≥1 miss)=1−pn → 1. No amount of instruction-writing removes the residual, because instructions are context, not enforcement.
+
The deterministic layer alone cannot reach reliability either. A decidable check bounds only the structural signal it was built to detect; semantic correctness is undecidable (§12), so ∏(1−cj)>0 for the semantic class. Without the instruction layer raising p, the factor (1−p) stays near 1 and the product is dominated by it.
+
Their composition is strictly better than either factor. Because 0<(1−p)<1 and each 0<(1−cj)<1, the product (5) is strictly smaller than (1−p) and strictly smaller than any single (1−cj). Reliability is the product of a soft factor and hard factors, and needs both kinds present.
+
+
□
+
+
+
+Equation (5) is the probability that the behaviour is both omitted by the agent (the independent event of probability 1−p) and undetected by every one of the k checks (each failing to catch with probability 1−cj, taken as conditionally independent given the miss). Claim 1: set k=0, the empty product is 1, so P=1−p; the n-task bound is the complement of n independent successes, pn. Claim 2: for the semantic-miss class every decidable cj is bounded below 1 (Rice's theorem: non-trivial semantic properties of programs are undecidable), so the product cannot vanish; with p not raised, (1−p) is near 1. Claim 3: multiplying a number in (0,1) by further numbers in (0,1) strictly decreases it below every factor. □
+
+
+
+
+Figure 1. The two-layer duality. The probabilistic instruction layer (Π3, purple) raises p by loading context but may drift (dashed arrows); the deterministic interception layer (Π2, teal) executes regardless of the model's choice and either passes the turn or blocks it (exit 2) back into the model for repair. The persistent store (Π1) feeds both. What escapes both layers is the residual (1−p)·∏(1−cj), handed to review or a later commit/CI gate. The whole sits inside a stewardship boundary (amāna, §9). Neither layer alone suffices — the formal content of the discipline never trust the output; earn trust with a check.
+
+
+
+
What the theorem says in one sentence
+
The practitioner's rule — never trust the probability engine's output; verify it — is the statement (1−p)>0. Theorem D turns that intuition into a design mandate: because the soft layer can never drive (1−p) to zero, a deterministic layer must exist to multiply it down; and because the hard layer can never catch the semantic class, the soft layer must exist to shrink what reaches it. The substrate is two-layered not by taste but by theorem.
+
+
+
5 The probabilistic failure model
+
+
Theorem D's equation (5) is worth reading as an engineering instrument, because it explains a lived experience and prices every design choice.
+
+
5.1 Why "it works, then forgets" is a certainty, not bad luck
+
With the instruction layer alone, the chance of at least one partial-work incident over n tasks is 1−pn. Even an excellent p=0.9 gives 65% after 10 tasks and 96% after 30. The agent that "usually remembers the docs" is, over a project's lifetime, near-certain to forget them at least once. The failure is geometric, so no degree of prompt-polishing escapes it — only a factor (1−cj) below 1 can bend the curve.
+
+
5.2 Why the deterministic gate is worth exactly one factor
+
Add one gate whose target is the decidable signal "code changed and no doc/state artifact changed." That signal is checkable in microseconds and c1≈0.95. With p=0.7, the per-task silent-miss rate falls from 30% to (1−0.7)·(1−0.95)=1.5% — a twentyfold reduction from a twenty-line hook. Crucially the class of surviving misses changes from "forgot the docs entirely" (structural, now caught) to "updated the docs imperfectly" (semantic, handed to review). The gate does not make the model think; it removes an entire failure mode from the model's shoulders.
+
+
5.3 The lattice of gates
+
The same classifier can run at three lifecycle points, and (5) shows their catches multiply: a turn-level Stop hook ⊂ a commit-level pre-commit hook ⊂ a PR-level CI job. Each later gate catches what earlier ones missed — the product (5) with k=3. This is also the answer to portability: where hooks are unavailable, the same deterministic check re-binds as a pre-commit hook or a CI step, moving the enforcement point without changing the mathematics.
+
+
+
The honest cost side
+
The model also prices the developer's real pain. Each silent miss costs a rework loop of expected size (1−p)·(1+r), where r is the re-explaining overhead that session-amnesia (F2) inflates. The architecture attacks both factors: gates convert silent misses into same-session fixes (r→0, caught before the developer sees the result), and persistence makes any residual loop cheap because the context is already standing. This is where the theory meets the user's stated grievance — wasted tokens, time, and quality — and answers it with a quantity, not a promise.
+
+
+
+
6 The unified algorithm set — the TASK loop
+
+
The faculties of Def. 4 are realized by seven algorithms. They are the reliability framework's A1–A7, recast here as the operations of the substrate: each is a faculty made mechanical, each binds to one lifecycle point, and together they form a single loop whose progress is guaranteed by an explicit worklist and whose floor is guaranteed by a deterministic gate.
+
+
A1 — IMPACT-CLOSURE (impact-awareness · simulate · before any code)
+
function IMPACT_CLOSURE(request):
+ Δ ← seeds(request) # named files + search hits
+ W ← Δ ; frontier ← Δ
+ while frontier ≠ ∅: # BFS to the fixpoint Δ* (Def. 3)
+ E ← grep_edges(I(frontier)) # R_discovered
+ ∪ map_edges(frontier) # R_declared
+ ∪ interface_edges(frontier) # callers, overrides, migrations, DI wiring
+ new ← E \ W ; W ← W ∪ new ; frontier ← new
+ check_constraints(W, DECISIONS) # flag conflicts, never override (I3)
+ return { (a, τ(a), action(a), risk(a)) : a ∈ W } # the impact table
+
Terminates in ≤|A| rounds (Thm. T5); on termination W ⊇ Δ* over discoverable+declared edges. This is simulate and impact of Def. 4, and it is exactly what the Impact Oracle prototype computes (§10).
+
+
+
A2 — EXECUTE-FULL (world-model · act on the whole closure)
+
implement every row of the impact table (code AND tests),
+ following conventions drawn from the store M (I3);
+partial implementation of the table = definitionally incomplete (Def. Done).
+
+
+
A3 — SYNC-VERIFY (self-correction · make every artifact true again)
+
function SYNC_VERIFY(changed):
+ I ← identifiers(diff(changed))
+ for a in DocSet ∪ map_artifacts:
+ hits ← grep(a, I)
+ if hits ≠ ∅: read(a); update stale content to match code
+ else: record verified-unaffected(a, "no mention of I") # checked, not assumed
+ if structure_changed: update ARCHITECTURE + diagrams + the map itself
+ if user_facing: append CHANGELOG
+ update STATE ; return {updated[], verified_unaffected[], doubts[]}
+
The verify operator: no artifact is ever declared unaffected without the check running — invariant I1's "∨ verified-unaffected" made mechanical. The doubts[] channel enforces I3 (surface ambiguity, do not guess).
+
+
+
A4 — HANDOFF (memory · write-back · session end)
+
function HANDOFF(K):
+ ensure SYNC_VERIFY ran
+ σ ← select(K, priority=[goal, next, decisions, gotchas, in_progress, done+why])
+ write STATE ← σ, |σ| ≤ B lines # REWRITE (bounded), not append
+ mirror durable decisions → DECISIONS # append-only log
+ if a convention was corrected → update CLAUDE.md / rules (self-maintenance)
+ propose commit # committing = portable memory
+
The bounded-compression checkpoint (|σ|≤B≈150) keeps the loader's cost O(B) forever — the snapshot+WAL pattern: STATE is the mutable snapshot, DECISIONS the durable log.
on SessionStart(startup | resume | clear):
+ record baseline b ← git HEAD # enables the gate's session diff (A6)
+ inject: STATE (≤8KB) + last 10 commits + uncommitted files + DoD reminder
+
The retrieve operator, made deterministic: continuity moves from "the agent may read the file" to "the context always contains it" — the same Π3→Π2 upgrade as the gate.
+
+
+
A6 — COMPLETION-GATE (self-correction · the deterministic floor · end of turn)
+
on Stop(session s):
+ C ← (diff base(s)..HEAD) ∪ worktree changes − internal bookkeeping paths
+ code ← C ∩ CodeClass \ DocClass \ .claude/ # regex-classified
+ docs ← C ∩ DocClass
+ if code ≠ ∅ ∧ docs = ∅: exit 2 + repair checklist + set marker # BLOCK once
+ else: exit 0 # ALLOW
+
The hard factor c1 of Theorem D. STATE counts as a doc artifact, so the weakest way to satisfy the gate is to update session state — which is exactly the continuity invariant I2. One check enforces a floor for both F1 and F2. Full decision table and safety proofs in §7.
+
+
+
A7 — KNOWLEDGE-ROUTER (M1 routing · where every fact lives)
+
route(f): # first match wins (total function, Thm. T6)
+ needed every session, stable → CLAUDE.md (always loaded, ≤200 lines)
+ relevant only to paths g → .claude/rules/x (paths: g; loads on match)
+ a procedure / workflow → skill (loads on invocation)
+ specialist's accumulated patterns → subagent memory
+ current work status → STATE.md (rewritten, injected)
+ decision + rationale → DECISIONS.md (append-only)
+
Keeps always-loaded context O(bounded) while total persisted knowledge grows without limit — the mathematical reason the substrate scales. This is the second face of routing: M1's complexity-router chooses a model tier by task difficulty; A7 chooses a storage home by knowledge type. Both are transparent and deterministic, and both reject an opaque-LLM classifier for the same reason — it would reintroduce the very (1−p) the hard layer exists to remove.
+
+
+
6.1 The loop
+
The seven compose into one meta-algorithm that fits every task, from a one-line fix to a multi-file feature:
Formally, iterate until the unsynced set U = { a ∈ Δ* : ¬updated(a) ∧ ¬verified(a) } = ∅ — a fixpoint loop whose progress is guaranteed by A1's explicit worklist and whose floor is guaranteed by A6. The soft stages (A1–A5, driven by instructions) raise p; the gate (A6, deterministic) guarantees the floor; persistence (A4/A5 over Π1) carries essential(K) across the session boundary.
+
+
+
+Figure 2. The unified TASK loop. Seven stages, each bound to its faculty, its algorithm (A1–A7), and its Qur'anic anchor (§9). The loop runs to the fixpoint U=∅ at the hub — no unsynced artifact remains. Soft stages raise p; the gate is the deterministic floor; the store carries state across the boundary. Rehydrate re-arms the baseline that the gate diffs against, closing the cycle across sessions.
+
+
+
+
7 The four invariants and the six correctness theorems
+
+
The TASK loop serves four invariants. Stated as logic, they port to any environment that supplies Π1–Π3.
+
+
+
Invariant
Statement
Kills
+
I1 — Consistency
∀(a,b)∈R: changed(b) in task t ⇒ updated(a) ∨ verified-unaffected(a) in the same t. "No artifact lies about the code."
F1
+
I2 — Continuity
At every session boundary: essential(K) ⊆ P, and the loader injects it. Corollary: everything essential lives in committed files — the only channel that crosses machines, terminals, web, and teammates.
F2
+
I3 — No fabrication
Every convention the agent acts on is derived from repo evidence, or documented in P, or asked — never sampled from priors. Unattended: assumptions are stated explicitly, never silent.
the amnesia root (§5)
+
I4 — Verified currency
Facts about the outside world (library versions, APIs) are checked against current sources at use time; the outcome is recorded. Training memory is a stale cache.
stale-knowledge drift
+
+
+
In Hoare-triple form, every task must satisfy
+
{ P consistent ∧ context loaded } execute(t) { Done(t) ∧ P consistent ∧ σ updated }
+
where the gate (A6) checks a decidable necessary condition of the postcondition and the algorithms construct it. I3 is the invariant the user named as the deepest problem — "the biggest problem is assumption." It is the direct architectural answer to the amnesia equation of §5: an under-specified input must be met with supplied context, a stated assumption, or a halt — never a silent guess drawn from the prior.
+
+
7.1 The correctness theorems
+
Six properties are proved of the deterministic layer. They are what make the hard factor of Theorem D trustworthy — a gate that could loop, brick a session, or miss its target signal would not earn its place.
+
+
T1 (No infinite loop; any stop sequence terminates in ≤2 attempts)
+
A block sets a per-session marker before exiting 2; marker creation is monotone. Any later attempt matches the marker row ⇒ ALLOW. Independently, the continuation carries stop_hook_active ⇒ ALLOW. Two independent guards, either sufficient; even if the process dies between marker-set and exit, the marker is already on disk. □
+
+
+
T2 (Fail-open: no hook failure can brick a session)
+
Every external call is guarded (|| true, 2>/dev/null, a jq→python3→empty fallback chain); every guard-failure path leads to a row whose decision is ALLOW. The only exit-2 path is the deliberate block row. The SessionStart hook only ever exits 0. □
+
+
+
T3 (Soundness of the block signal — no false silence)
+
On the first stop of a session with usable git: C is the union of the baseline diff and an untracked-inclusive worktree scan, so every changed path is in C; classification is a total function of path; the row order reaches the block row exactly when code changed and no doc changed. Hence a silent code-only completion is impossible at the session's first completion — the agent must fix the docs or explicitly justify and update STATE, both visible to the developer. □
+
+
+
T4 (Continuity under handoff)
+
If A4 ran and its commit is pushed/pulled, then for any next session on any machine the loader injects σ at start (A5 reads the committed file), so essential(Ki) ∩ σ ⊆ Ki+1(0). Residual risk is exactly selection error in the handoff (what it chose not to write), bounded by the priority order and by DECISIONS catching the durable class. □
+
+
+
T5 (Closure termination, A1)
+
A monotone worklist on the finite set A: each round adds ≥1 artifact or stops, so ≤|A| rounds. This is the finite-lattice Kleene fixpoint of Def. 3. □
+
+
+
T6 (Router totality, A7)
+
The routing chain ends in catch-alls per scope; every fact matches ≥1 arm; first-match makes the assignment unique. Hence route is a total function — every piece of knowledge has exactly one home, which is what keeps the always-loaded budget bounded. □
+
+
+
+
How the theorems earn Theorem D
+
Theorem D says reliability needs a deterministic factor with cj→1 on its target signal. T3 is precisely that guarantee (the block fires exactly on the target signal); T1 and T2 ensure the factor is safe to add (it never loops, never bricks); T5 and T6 ensure the soft-layer machinery it composes with is well-defined (the closure terminates, the router is total); T4 extends the guarantee across the session boundary that F2 attacks. The six local proofs are what make the one global theorem deployable rather than merely true.
+
+
+
8 The crosswalk — one object, three vocabularies
+
The table below is the operational proof of the paper's claim: every concept appears in all three vocabularies, and the final column states the relationship that makes them one object. Three rows (marked ●) are not analogies but identities — the same mathematics under two names. The full machine-readable crosswalk is a companion artifact.
+
Notation reconciliation — the P1/P2/P3 collision
+
Both source frameworks independently use the labels P1/P2/P3. In the substrate paper they are model properties (P1 statelessness, P2 frozen weights, P3 bounded context, §2); in the reliability framework they are the three primitives (persistent store, lifecycle interception, instruction channel). This paper keeps P1–P3 for the model properties and renames the primitives Π1/Π2/Π3 throughout (§3.1). Every reference to a primitive in this paper is written Πn.
the agent/model whose behavior instructions can only raise the PROBABILITY of (p<1)
Claude / Codex / Cursor / Gemini / Aider — the model the kit wraps, never modifies
identical object: all three treat the model as a fixed probabilistic map to be wrapped, not retrained
+
2
Impact-awareness / partial-work failure ●
Faculty: impact-awareness gap; the developer silently simulates 'what will this edit break'. Prototype I = Impact Oracle (reverse-dependency blast radius with confidence decay)
F1 partial work; dependents operator N(X); required change closure Delta* = least fixpoint of X |-> X ∪ N(X); Done predicate; Algorithm A1 IMPACT-CLOSURE
IDENTICAL MATHEMATICS. The Impact Oracle computes N(X) by reverse reachability on the dependency graph; Delta* is exactly the oracle's transitive blast-radius set. The oracle adds a confidence-decay weight the framework's set-theoretic N(X) leaves boolean.
+
3
Memory / session-amnesia failure
Faculty: persistent memory gap; each context window is ephemeral. Validity-anchored memory (facts carry confirmed/discredited state updated by verified outcomes)
F2 session amnesia; continuity invariant I2 (essential(K_i) ⊆ P and loader L must load it); handoff operator H; K_{i+1}(0)=L(P)
Same failure, same fix (external bounded-compression checkpoint). Substrate adds validity-anchoring (prune by whether a past prediction was externally confirmed); framework adds the snapshot+WAL split (STATE mutable vs DECISIONS append-only).
+
4
Why assumptions happen (the root the user named) ●
M2 assumption/uncertainty gate: under-specified input -> the model confabulates a convention
amnesia equation: when f ∈ essential(K) is missing from L(P), assumption ≈ argmax P(convention | training data) — the mathematically EXPECTED result of missing context, not misbehavior. Invariant I3 (no fabrication)
The framework gives the substrate's M2 gate its formal justification: assumption is argmax over the prior, so the fix is to supply context (raise what L loads) OR gate on missing context (halt), never to hope.
Algorithm A6 COMPLETION-GATE (deterministic Stop-hook floor); Hoare postcondition Done(t); the verification operator in A3 (verified-unaffected requires an actual grep, not an assumption)
docs-guard.sh Stop hook (blocks finish if code changed but no doc/state artifact did); reviewer agent verdict
Same verify-gate. Substrate framed it as a mechanism; framework proves it terminates (T1) and is sound (T3) and makes it the deterministic layer of the duality.
Two facets of one routing principle. Substrate M1 routes by TASK COMPLEXITY -> model tier (cost). Framework A7 routes by KNOWLEDGE TYPE -> storage home (context budget). Both are transparent/deterministic, both reject an opaque-LLM classifier for the same reason.
+
7
Task decomposition
M3 task/session decomposition
HTN closure -> ordered task list (A1 output is the worklist); the meta-algorithm TASK(t)
the typed artifact graph (A, tau, R = R_declared ∪ R_discovered)
documentation-map.md + the repo itself + ARCHITECTURE.md
Same graph. Substrate builds it from AST for code; framework generalizes the node type to {code,test,doc,config,diagram} so DOCS become first-class dependents — the key generalization that turns 'impact on code' into 'impact on all artifacts'.
+
11
Continual learning from outcomes
Faculty: learning without touching theta (non-parametric always-on + parametric LoRA/EWC)
Reflexion loop made cross-session (I3/I4); DECISIONS.md as precedent DB; A4 mirrors durable lessons
Reflexion rule in CLAUDE.md; STATE gotchas; agents' memory: project
Same 'learn without retraining' stance. Both store verbal lessons externally and re-inject; substrate adds the parametric option, framework keeps it purely non-parametric (committed files).
+
12
Rehydration (session start)
closed-loop write-back/read-back band in the substrate architecture
Algorithm A5 CONTEXT-REHYDRATE; loader L; records git baseline for the gate
The read side of the substrate's memory loop, made deterministic (inject, don't hope the agent reads).
+
13
Independent verification / judge
self-correction faculty; the honest-negative-result caution (models correct poorly alone)
LLM-as-Judge applied: reviewer agent (fresh context, explicit criteria, adversarial); self-consistency for critical changes (majority of N)
reviewer.md agent; DoD item 7
Substrate cited the finding that self-correction alone is weak; framework operationalizes the fix (an EXTERNAL judge), which is the same 'trust only an external check' principle as the gate.
+
14
The two-layer duality (THE central new insight) ●
implicit: the substrate wraps a probabilistic core with deterministic external structure, but v2 never states it as a law
DESIGN LAW: instructions (Pi3) are PROBABILISTIC (raise p); interception (Pi2) is DETERMINISTIC (guarantee a floor c->1). P(silent miss) = (1-p)·prod_j(1-c_j). Since p<1 always, neither layer alone suffices.
the split itself: CLAUDE.md/rules/skills = soft layer; hooks (docs-guard, session-context, intent-router) = hard layer
THIS is what the kit gives the substrate theory: the reason the substrate must be TWO layers, stated as a theorem. The user's own discipline ('never trust AI output — it is calculated probability') is exactly (1-p)>0, which forces the deterministic layer.
+
15
The probabilistic failure model
eval honesty: perfect accuracy shows separation not a benchmark; p<1
P(>=1 miss) = 1 - p^n over n tasks (0.9 -> 65% at 10, 96% at 30); layered: P(silent miss)=(1-p)·prod(1-c_j)
the lattice of gates: turn-level (hook) ⊂ commit-level (pre-commit) ⊂ PR-level (CI)
The math that makes 'it works sometimes then forgets' a geometric certainty rather than bad luck — and quantifies why each added deterministic gate multiplies down the residual.
+
16
Stewardship / governance boundary
STEWARDSHIP/amana wrapper (33:72) around the whole architecture
amana in I3/I4 as no-fabrication + verified-currency; least privilege, reversibility, logged rationale, scope-boundedness
The ethical frame the substrate draws from the lens is realized as concrete safety properties in the framework (fail-open, reversibility) and kit (auditable hooks).
+
+
The three anchor identities
+
Impact-Oracle blast-radius ≡ change-closure Δ*. Reverse reachability on the dependency graph is the least fixpoint of X ↦ X ∪ N(X). The oracle weights it with confidence decay; the framework’s N(·) is its boolean core.
+
M2 assumption gate ≡ the amnesia equation.assumption ≈ argmax P(convention | training) is why an under-specified prompt is answered with a confabulated convention — so the gate supplies the missing context or halts; it never hopes.
+
The substrate’s two layers ≡ the design law. Instructions raise p<1; interception guarantees a floor c→1. Theorem D. This is the formal statement of the governing discipline.
+
+
9 The Qur’anic epistemology — the full fourteen mappings
+
+
How the lens is used, and how it is not
+
The Qur’an is used here as a framing lens and ethics source, never as technical authority for an engineering claim. No verse proves that an algorithm works or a data structure is correct — those stand or fall on their engineering merits alone (§4–§7). What the lens supplies is (1) a vocabulary for the agent’s epistemic obligations — what it owes to truthfulness, verification, and stewardship; (2) a hierarchy of knowledge (‘ilm → fahm → ḥikma) that motivates a layered memory rather than a flat store; and (3) ethical constraints on autonomy (amāna, tabayyun) that translate into concrete safeguards. A mapping marked load-bearing directly motivates a specific architectural decision — e.g. a mandatory gate, not an optional one; a mapping marked metaphor is illustrative. Even load-bearing mappings must be independently defensible: the verse explains why we insist, not that it works.
+
All Arabic is the clean Uthmani-script edition and all translations are Abdel Haleem, retrieved verbatim via the quran.ai connector; tafsir references are Ibn Kathir. No Qur’anic text is reproduced from model memory.
+
+
The fourteen mappings are the epistemology behind the architecture: each names an obligation that a probabilistic engine, left alone, will not honour, and which the deterministic layer therefore exists to enforce. Twelve are load-bearing; two are metaphor. They are grouped by the faculty they govern.
“Do not follow blindly what you do not know to be true: ears, eyes, and heart, you will be questioned about all these.”
+
Before any mutation, the agent must run a pre-action check of what will be affected, whether it has sufficient context, and whether the predicted outcome is evidence-supported rather than pattern-matched. This is anchor identity #1: the impact closure Δ* (A1) computed before coding. Ibn Kathir glosses the verse (via Qatadah) as a prohibition on claiming knowledge one lacks — for the agent: do not call a file safe to modify without reading it, nor a test passing without running it. The verse’s ‘ears, eyes, and heart … questioned’ maps to the audit trail: every channel used is logged so the decision can be reconstructed.
+
+
9.2 Self-correction — verify, reflect, iterate
+
+
Q 49:6 — fa-tabayyanū (the verification gate) load-bearing
“Believers, if a troublemaker brings you news, check it first, in case you wrong others unwittingly and later regret what you have done,”
+
The operative term tabayyun demands active investigation, not passive acceptance — a mandatory step between receiving information and acting. This is the completion gate A6 and the external verifier of Def. 4: before applying a fix based on an error report, a request, or its own diagnosis, the agent re-reads the file and re-confirms the error. The gate is architectural (a deterministic pipeline step), not advisory — the verse’s command is categorical, exactly the hard factor c→1 of Theorem D.
“Will they not think about this Quran? If it had been from anyone other than God, they would have found much inconsistency in it.”
+
‘They would have found much inconsistency’ makes internal contradiction evidence of flawed origin — the argument behind self-consistency checking (SYNC-VERIFY’s doubts[] channel, and the majority-of-N escalation of the reviewer). If a planned change contradicts the agent’s stated reasoning or the tests it just read, the metacognitive pass flags it and halts.
“Will they not contemplate the Quran? Do they have locks on their hearts?”
+
The ‘locks on hearts’ image maps to a real failure mode: when context saturates, the agent becomes functionally unable to reconsider. The metacognitive controller must be able to reset the working context and re-examine from a fresh framing — structured backtracking, the fresh-context reviewer that never inherits the builder’s narrative.
“(2:31) He taught Adam all the names [of things], then He showed them to the angels and said, ‘Tell me the names of these if you truly [think you can].’ (2:32) They said, ‘May You be glorified! We have knowledge only of what You have taught us. You are the All Knowing and All Wise.’”
+
‘He taught Adam all the names’ — knowledge begins with naming entities and their relations. This is the typed artifact graph (Defs. 2–3): not a flat file listing but a semantic map of which function calls which, which test covers which class. The angels’ ‘we have knowledge only of what You have taught us’ is precisely the model’s situation — it knows only its context window; the external world-model supplies the names of entities that exceed it.
+
+
9.4 Continual learning — knowledge as ongoing increase
“exalted be God, the one who is truly in control. [Prophet], do not rush to recite before the revelation is fully complete but say, ‘Lord, increase me in knowledge!’”
+
‘Do not rush … but say, increase me in knowledge’ — two mechanisms at once. The prohibition on rushing before revelation is complete is the halt-on-incomplete-context rule (I3). The prayer for increase is the cross-session experience store that grows priors without touching θ (P2): learning is external memory that changes what the agent sees next, not a weight update.
“What about someone who worships devoutly during the night, bowing down, standing in prayer, ever mindful of the life to come, hoping for his Lord’s mercy? Say, ‘How can those who know be equal to those who do not know?’ Only those who have understanding will take heed.”
+
‘How can those who know be equal to those who do not?’ establishes that knowledge is not fungible with ignorance — motivating the architecture’s distinction between grounded mode (relevant prior experience loaded) and ungrounded mode (base model alone), and surfacing that distinction to the user rather than hiding it.
“(96:1) Read! In the name of your Lord who created: (96:2) He created man from a clinging form. (96:3) Read! Your Lord is the Most Bountiful One (96:4) who taught by [means of] the pen, (96:5) who taught man what he did not know.”
+
‘Who taught by the pen’ — al-qalam is the instrument of externalization, turning ephemeral thought into durable record. Every context window is ephemeral (unwritten thought), so the persistent store (Π1) is ‘the pen’: write-back (A4) captures what is learned, retrieval (A5) reloads it next session, consolidation organizes raw experience into structured knowledge. ‘Taught man what he did not know’ — the pen does not merely record; it grants access beyond unaided capacity.
“We offered the Trust to the heavens, the earth, and the mountains, yet they refused to undertake it and were afraid of it; mankind undertook it- they have always been inept and foolish.”
+
An agent that can modify a codebase bears a trust. The verse’s structure is decisive: the heavens refused the trust, recognizing its weight; the human bore it and was called ẓalūman jahūlā (given to wrongdoing and ignorance). The design is therefore dual: (a) the agent operates within explicit authorization — it may not exceed asked scope (M5, anti-over-engineering); (b) the architecture assumes the agent will err and builds in rollback, sandboxing, and incremental commit as structural safeguards. Ibn Kathir’s gloss (Ibn Abbās) ties amāna to accountability — outcome-linked feedback: actions must be traceable to outcomes, and outcomes feed the learning store. This is the governance boundary around Figures 1 and 2, realized concretely as fail-open safety (T2), reversibility, and auditable hooks.
+
+
9.7 The four governing concepts
+
Beyond the verses, four Qur’anic concepts structure the whole architecture, each load-bearing:
+
+
‘ilm → fahm → ḥikma (knowledge → understanding → wisdom) — the epistemological hierarchy that mandates a layered memory: raw logs (‘ilm) at the base, a consolidation pass extracting patterns (fahm), a decision-support layer applying them (ḥikma). Dumping everything into one flat vector store collapses the hierarchy — this is the architectural argument for STATE vs DECISIONS vs ARCHITECTURE as distinct stores with distinct update rules, not one file.
+
ḥifẓ + murāja‘a (preservation + spaced review) — memory is not write-once: a maintenance cycle reinforces recurring patterns, decays stale entries, and resolves contradictions. This is the memory-scoring rule (importance × recency-decay) and the validity-anchoring of Def. 4.
+
tabayyun (verification before action) — the categorical gate of §9.2, the hard factor of Theorem D.
+
tadabbur (structured reflection on consequences; root d-b-r, ‘what follows’) — the metacognitive controller as a trace-forward through the dependency graph: what comes after this change, what breaks, what it assumes. This is A1’s consequence simulation, not a vague ‘think again’.
+
+
+
The lens in one line
+
Every faculty the model lacks corresponds to an obligation the tradition names: lā taqfu (do not act without knowledge) → impact-closure; tabayyun (verify the report) → the gate; ta‘līm al-asmā’ (know the names) → the world-model; al-qalam (the pen) → persistent memory; rabbi zidnī ‘ilmā (increase me in knowledge) → continual learning; al-amāna (the trust) → the stewardship boundary. The engineering says how; the lens says why it is owed.
+
+
+
10 The prototypes — what measurement did to the theory
+
+
+
This section reports a refutation of the claims made in earlier editions of this work
+
Earlier editions reported that the impact oracle achieved perfect recall (R = 1.00)
+and that the router/gate pair achieved a perfect separation with a 62.1% cost saving. Both results came
+from self-built demonstration sets. Under a protocol frozen before data collection and run against nine
+real open-source repositories and eighty real issues, both claims failed. They are
+retained here, corrected, because the failure is the most useful thing this work produced.
+
+
+
10.1 Prototype I — the impact oracle, refuted and repaired
+
+
Ground truth was mined from real commit history: two files are impact-related if they changed in the
+same non-merge commit, under filters fixed in advance. Against 801 labelled files in nine repositories:
+
+
+
Method
Precision
Recall
F1
+
Impact oracle, as shipped
0.398
0.022
0.042
+
grep baseline (no graph)
0.353
0.573
0.437
+
+
+
Recall was 0.022, not 1.00 — a gap of roughly
+45×. Ninety-one percent of files received exactly zero recall, and a textual baseline with no
+dependency graph at all outperformed the graph-based oracle by an order of magnitude on F1.
+
+
+
What this does to Theorem T5 — the correction that matters
+
Earlier editions argued that perfect recall follows from T5: a reverse-reachability closure run
+to fixpoint cannot miss a reachable dependent. The theorem is not wrong; the inference from it was. T5
+guarantees completeness relative to the relationR over which the closure
+is computed. It says nothing about whether R contains the edges that matter, nor
+whether the graph was built correctly. Both failed here. Classifying all 18,221 false negatives showed
+94.7% were sibling pairs — two files sharing a dependency, reachable only by a mixed
+forward-then-reverse path that a reverse-only traversal cannot take — and a construction defect
+silently produced phantom nodes for one of the two standard Python package layouts. Only 3.2% were beyond
+any static path.
+
The general lesson for the formalism: a completeness theorem over a relation transfers no
+guarantee to practice until the relation is shown to be adequate and its construction is verified.
+Def. 3's Δ* is sound; the deployed R was not
+the R the theory assumed.
+
+
+
Because 96.8% of the missed impact was diagnosed as reachable in principle, we repaired both defects and
+re-ran the same frozen protocol, with parameters fixed on six repositories and frozen before three held-out
+repositories were touched:
+
+
+
Configuration
Precision
Recall
F1
+
As shipped
0.398
0.022
0.042
+
+ construction fix
0.304
0.242
0.270
+
+ sibling/forward relations, held-out
0.305
0.653
0.416
+
grep baseline, held-out
0.269
0.601
0.371
+
+
+
The repaired oracle beats the baseline for the first time, reaching 66.8% of the achievable ceiling. The
+margin is narrow and the intervals are close. Two details are worth more than the headline. First, the
+obvious repair of the construction defect is unsafe — it fabricates dependency edges through
+standard-library name collisions — so we applied a more conservative fix with a smaller gain
+(11.0× rather than 14.5×); a tool that invents edges to raise recall is worse than one that misses
+them. Second, because the tuning/held-out split was declared in advance, we could see that precision
+generalised far worse than recall (−0.141 against −0.021). Tuning on all nine repositories would
+have produced a better number and concealed that fact.
+
+
+
+Figure 3. The repair sequence with 95% bootstrap confidence intervals, against
+the grep baseline (diamonds) and the static ceiling. Both fixes are necessary; neither alone crosses the
+baseline.
+
+
+
10.2 Prototype II — the router and gate, refuted
+
+
Re-evaluated on eighty tasks drawn from real GitHub issues and pull requests, with every threshold frozen
+at its tuned value:
+
+
+
Metric
Tuned set (n=30)
Held-out (n=80)
+
Gate F1 (should-ask)
1.00
0.37
+
Gate recall
1.00
0.31
+
Routing, exact tier
1.00
0.53
+
Routing, within one tier
1.00
0.91
+
Cost saving — first attempt, ungated
62.1%
59.5%
+
Cost saving — first attempt, correctness-gated
—
3.6%
+
Cost saving — total pipeline spend, ungated
—
−20.2%
+
Cost saving — total pipeline spend, gated
—
2.8%
+
+
+
The gate missed roughly seven in ten under-specified requests. Routing retained partial signal —
+within-one-tier accuracy of 0.91 is well above chance, so the complexity rubric measures something
+— but exact-tier accuracy fell to 0.53, and the cost saving did not merely shrink but inverted: routing does save 59.5% in raw dollars on first attempts alone, but almost none of that cheaper output is correct (3.6% once gated), and counting what the pipeline actually spent escalating up the tier ladder, it costs 20.2% more than always using the premium tier. Labelling noise is real and reported rather than hidden: inter-rater agreement on the
+should-ask label was κ = 0.52, moderate, which bounds how well any gate could score here.
+
+
+
What the earlier numbers actually measured
+
Nothing about the demonstrations was fabricated; every number in the earlier editions was real. They
+measured the wrong thing. A rubric scored against the thirty tasks its thresholds were tuned on reports the
+fit, not the capability, and a blast radius measured against mutations of a package the authors wrote
+reports the authors' model of impact, not the codebase's. The mechanism was never the weak link in the
+evidence; the evidence standard was.
+
+
11 forgekit — the deployed binding
+
The theory is implementation-independent; forgekit is one binding of it, and its precursor claude-e2e-kit is the reference realization on Claude Code. The mapping is exact:
The kit grounds its mechanisms in the same literature the substrate's faculties cite — CoALA's four memories, MemGPT's paging, Generative Agents' recency×importance scoring, Reflexion's verbal reinforcement, ReAct's reason–act interleaving, LLM-as-Judge's independent evaluator, AlphaCodium's phase-gated flow, Voyager's growing skill library, and the classical decision loops (BDI, OODA/PDCA, HTN). It is the same architecture, cited from the same shelf, and shipped. The two prototypes of §10 slot in as the mechanical cores of /impact and the effort-router.
+
+
+
Why one binding matters for the theory
+
That an independent team, starting from production failures rather than from cognitive faculties, built the same seven algorithms and stated the same design law is the strongest available evidence that the architecture is discovered, not invented — a convergent solution to a structural problem, the way distributed systems converge on snapshot+WAL. The synthesis does not merge two guesses; it records a convergence.
+
+
+
12 Honest limits — what no architecture can guarantee
+
+
Semantic correctness is undecidable. The gate proves "a doc artifact changed", not "the docs are now true"; A3's grep proves "mentions were visited", not "the prose is accurate". By Rice's theorem the last layer is unavoidable — tests for behaviour, human review for meaning. The architecture's job is to make that review cheap: everything arrives already-attempted, with an updated / verified / doubts report.
+
The soft layer's p is real but bounded. Theorem D quantifies the residual; it does not abolish it. A determined agent can satisfy the letter of a gate (touch STATE with one line) without its spirit — which is why no single layer is trusted, by design.
+
Rdiscovered misses unnamed coupling. A doc that describes behaviour without naming any identifier has no grep edge; such couplings must be lifted into Rdeclared (the documentation map) — exactly what that table is for.
+
One prototype is not five faculties. The substrate prototypes instantiate impact-awareness and routing/gating well; memory, learning, and imagination remain the harder research frontier, and the honest ecosystem map marks the assumption gate (M2) and outcome-based learning as genuine whitespace the current stack does not fill.
+
The lens is framing, not proof. Every Qur'anic mapping motivates a design choice; none of them validate one. The engineering in §4–§7 stands on its own or not at all — the tradition tells us why a safeguard is owed, never that it works.
+
+
+
13 Conclusion
+
A language model that writes code is a fixed probabilistic map, and three independent efforts — one from cognition, one from production failures, one from a shipped codebase — converged on the same remedy: wrap it in an external, stateful architecture that supplies the faculties it structurally lacks. This paper showed they describe one object. The impact-awareness faculty is the change-closure fixpoint; the assumption gate is the amnesia equation; and both rest on a single theorem — reliability is the product of a probabilistic instruction layer that raises p<1 and a deterministic interception layer that guarantees a floor, with neither alone sufficient.
+
+
The limit this edition discovered the hard way
+
The honest limits listed below were all stated before any real-repository measurement existed. One more
+belongs at their head, and it subsumes several: a self-built demonstration can overstate a
+mechanism's field performance by more than an order of magnitude, and no amount of careful caveating of a
+demonstration converts it into evidence. Both prototypes here were labelled honestly as
+demonstrations, with their tuning disclosed. That disclosure was accurate and it was insufficient —
+readers, including their authors, carried the numbers forward as if they bounded real behaviour. The
+remedy is not better hedging but a different evidence standard: ground truth the authors did not construct,
+a protocol frozen before the data is seen, and a split declared before any parameter is fitted.
+
+
+
That theorem is the formal content of a plain discipline: the output of a probability engine is never to be trusted on its own; trust is earned by an external check. The Qur'anic lens gives that discipline its oldest names — lā taqfu, do not pursue what you do not know; tabayyun, verify the report before you act; al-amāna, the weight of a trust accepted by one who may err. The mathematics says how to build the check. The tradition says why it is owed. The codebase shows it runs.
+
Companion artifacts: the three-way crosswalk (JSON + markdown), the graded reference set (Appendix A), and two runnable prototype packages (impact-oracle, router-gate). This synthesis consolidates and does not supersede the v2 Theory → Evidence → Build-Map edition, which carries the empirical evidence layer and the full ecosystem map.
+
+
Appendix A — Graded reference set (new sources)
+
The synthesis draws in a body of cognitive-architecture and process literature beyond the substrate paper's original 32 references. Each new source was independently verified this pass — modern arXiv sources by direct metadata fetch, classical works by primary-host search or established secondary knowledge — and graded: confirmed (record retrieved, attribution matches), traceable (the work clearly exists and is correctly attributed, but rests on established secondary knowledge rather than a single retrievable record), unverifiable (could not confirm). The tally: 8 confirmed, 6 traceable, 0 unverifiable.
+
+
Source
ID
Grade
Note
+
Cognitive Architectures for Language Agents Theodore R. Sumers, Shunyu Yao, Karthik Narasi, 2023
2309.02427
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Unifies memory, planning/reasoning, action, and learning modules into a single CoALA framework for language agents, giving the cognitive-substrate work's memory/im…
+
ReAct: Synergizing Reasoning and Acting in Language Shunyu Yao, Jeffrey Zhao, Dian Yu et al., 2022
2210.03629
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Interleaves reasoning traces with actions in a single LLM prompt loop, the foundational pattern the cognitive-substrate and reliability-framework agent loops build on.
+
A Survey on LLM-as-a-Judge Jiawei Gu, Xuhui Jiang, Zhichao Shi et al., 2024
2411.15594
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Surveys the emerging practice of using LLMs themselves as evaluators/judges, directly relevant to any impact-awareness or self-correction mechanism that relies on an LL…
+
Code Generation with AlphaCodium: From Prompt Engine Tal Ridnik, Dedy Kredo, Itamar Friedman, 2024
2401.08500
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Replaces single-shot prompting with an iterative 'flow engineering' test-generate-fix loop for code generation, an applied precedent for the reliability frame…
+
SWE-agent: Agent-Computer Interfaces Enable Automate John Yang, Carlos E. Jimenez, Alexander Wettig, 2024
2405.15793
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Introduces an agent-computer interface (a constrained action/observation space) purpose-built for LM agents doing software engineering, directly relevant to forgekit&#x…
+
Voyager: An Open-Ended Embodied Agent with Large Lan Guanzhi Wang, Yuqi Xie, Yunfan Jiang et al., 2023
2305.16291
confirmed
Retrieved via arXiv metadata API; title/authors match claim exactly. Demonstrates a lifelong-learning embodied agent that maintains and grows a skill library over time, the clearest existing analogue to persistent, growing external memor…
+
Agentic AI in the Software Development Lifecycle: Ar Happy Bhati, 2026
2604.26275
confirmed
Retrieved via arXiv metadata API; the ID resolves to a REAL paper dated 2026-04-29 (April 2026, which is in the past relative to today, 2026-07-11 — so 'future-dated' only relative to the cited work's original claim date, …
+
Agent-as-a-Judge Runyang You, Hongru Cai, Caiqi Zhang et al., 2026
2601.05111
confirmed
Retrieved via arXiv metadata API; ID resolves to a REAL paper dated 2026-01-08. Title is exactly 'Agent-as-a-Judge' and abstract frames it as 'the first comprehensive survey' of the Agent-as-a-Judge paradigm, matching…
The founding Agent-as-a-Judge paper that coined the term; added on the verification track's explicit recommendation to disambiguate from the 2026 survey (2601.05111).
+
HTN Planning: Complexity and Expressivity Erol, Hendler, Nau, 1994
—
traceable
Classical AI-planning paper, no arXiv/OpenAlex record. Verified via live web_search (this turn) against AAAI's own paper page, Semantic Scholar, and a downstream paper's reference list (arXiv:1403.7426, 'An Overview of Hie…
+
BDI Agents: From Theory to Practice Rao, Georgeff, 1995
—
traceable
Classical agent-architecture paper, no DOI/arXiv record. Verified via live web_search (this turn) against the AAAI-hosted PDF (cdn.aaai.org/ICMAS/1995/ICMAS95-042.pdf), gabormelli.com/RKB, and multiple independent downstream reference li…
+
Thinking, Fast and Slow Kahneman, D., 2011
—
traceable
Classical trade/academic book, not indexed on arXiv or as a journal article with a DOI in the usual sense; existence and content (System 1 / System 2 dual-process framing) are well-established general knowledge, not independently re-veri…
+
Über das Gedächtnis: Untersuchungen zur experimentel Ebbinghaus, H., 1885
—
traceable
Foundational 1885 monograph establishing the forgetting curve; predates modern indexing entirely, existence is well-established historical/secondary knowledge, not independently re-verified against a bibliographic API in this pass.
No formal published paper exists — Boyd never formally published the OODA loop in a journal; it survives via briefing-slide decks and secondary military-strategy literature. Rubric correctly flags this as having 'no formal paper.&#x…
Management/quality-control doctrine spanning multiple books across decades, not a single citable paper. Existence and attribution (Shewhart origin, Deming popularization) are well-established secondary knowledge, not independently re-ver…
+
+
+
The two future-dated identifiers — verified, not assumed
+
Two IDs in the source material are dated 2026. Both resolve to real preprints whose content matches the claim, confirmed by direct arXiv fetch rather than by topic plausibility. arXiv:2604.26275 ("Agentic SDLC", Apr 2026) is a genuine but single-author, non-peer-reviewed preprint — cited as a recent preprint claim, not an established result. arXiv:2601.05111 ("Agent-as-a-Judge", Jan 2026) is a real survey, but is a different paper from the founding work that coined the term — Zhuge et al. 2024 (arXiv:2410.10934), which is the reference this paper uses for the independent-judge concept in §11. Both are recorded here so the distinction is not lost.
Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan et al. (2023). Cognitive Architectures for Language Agents. arXiv:2309.02427.
+
Ebbinghaus, H. (1885). Über das Gedächtnis: Untersuchungen zur experimentellen Psychologie (Memory: A Contribution to Experimental Psychology). Leipzig: Duncker & Humblot (book/monograph).
+
+
Learning without weight updates
+
+
Kirkpatrick et al. (2017). Overcoming catastrophic forgetting in neural networks. PNAS / arXiv:1612.00796.
+
Sun et al. (2020). Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. arXiv:1909.13231 / ICML 2020.
+
Ba, Hinton, Mnih, Leibo, Ionescu (2016). Using Fast Weights to Attend to the Recent Past. arXiv:1610.06258.
+
Hu et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
+
von Oswald et al. (2022). Transformers Learn In-Context by Gradient Descent. arXiv:2212.07677.
+
Dai et al. (2022). Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers. arXiv:2212.10559.
+
Guanzhi Wang, Yuqi Xie, Yunfan Jiang et al. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291.
+
+
Reasoning, verification & judgement
+
+
Cobbe et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168.
+
Lightman et al. (2023). Let's Verify Step by Step. arXiv:2305.20050.
+
Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. arXiv:2303.11366.
+
Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651.
+
Wang et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
+
Huang et al. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798.
+
Shunyu Yao, Jeffrey Zhao, Dian Yu et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.
+
Jiawei Gu, Xuhui Jiang, Zhichao Shi et al. (2024). A Survey on LLM-as-a-Judge. arXiv:2411.15594.
Ferrante, Ottenstein, Warren (1987). The Program Dependence Graph and Its Use in Optimization. ACM TOPLAS.
+
+
Agent process & software lifecycle
+
+
Tal Ridnik, Dedy Kredo, Itamar Friedman (2024). Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv:2401.08500.
+
John Yang, Carlos E. Jimenez, Alexander Wettig et al. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. arXiv:2405.15793.
Boyd, J.R. (None). OODA Loop (Observe-Orient-Decide-Act). Unpublished briefings/lectures (e.g. 'A Discourse on Winning and Losing', unpublished collection of briefing slides, c. 1976-1996); no single peer-reviewed paper.
+
Shewhart, W.A. (originator); Deming, W.E. (popularizer) (None). PDCA / Plan-Do-Check-Act cycle (the 'Shewhart Cycle', popularized by Deming). Originates in Shewhart, Statistical Method from the Viewpoint of Quality Control (1939); popularized in Deming, Out of the Crisis (1982) and later works.
+
Happy Bhati (2026). Agentic AI in the Software Development Lifecycle: Architecture, Empirical Evidence, and the Reshaping of Software Engineering. arXiv:2604.26275.
+
+
+
Appendix B — The crosswalk artifact
+
The full three-way crosswalk (§8) is provided as a machine-readable companion in two forms: crosswalk.json (structured, with the notation reconciliation and the three anchor identities) and crosswalk.md (readable table). Together with this paper, the two graded map files from the v2 edition (evidence map, ecosystem map), and the two runnable prototype packages (impact-oracle, router-gate), they form the complete synthesis deliverable set.
+
+
+A Formal Theory of the Cognitive Substrate for Coding Agents — synthesis edition.
+Unifying the substrate faculties, the end-to-end reliability framework, and the forgekit implementation.
+The mathematics says how the check is built; the tradition says why it is owed; the codebase shows it runs.
+