research: full research programme + pre-registered evaluation refuting two reliability mechanisms - #147
Merged
Conversation
…ty mechanisms Adds research/empirical-refutation/: a pre-registered empirical evaluation of the impact oracle and the complexity-router/assumption-gate against real repositories and real issues, which REFUTES both of their previously reported headline claims. Findings: - Impact oracle recall 1.00 (self-built demo) -> 0.022 (9 OSS repos, 801 files). A grep baseline with no dependency graph beats it ~10x on F1. - Router/gate F1 1.00 -> 0.37 on 80 real issues/PRs, thresholds frozen. - Cost saving does not shrink but inverts: counting the pipeline's actual escalation spend, routing costs 20.2% MORE than always-premium. Diagnosis and repair: - 96.8% of missed impact was reachable in the graph the tool already builds: 94.7% sibling pairs unreachable by reverse-only traversal, plus a construction defect affecting src-layout packages. Only 3.2% was a true proxy ceiling. - Repairing both, with parameters frozen before three held-out repos were touched, lifts recall 0.022 -> 0.653 and F1 0.042 -> 0.416, beating grep (0.371) for the first time. The margin is narrow and precision generalised worse than recall. Also corrects a theoretical claim: perfect recall was inferred from Theorem T5, but T5 guarantees completeness only relative to the relation the closure runs over. It says nothing about whether that relation contains the edges that matter. Includes the frozen pre-registration, mined ground truth, held-out task set with two labelling passes, all results with bootstrap CIs, both prototype versions, a replication package, and our own adversarial review listing four numbers we had stated incorrectly before correction.
… retries The held-out correctness judge is the same model as the tier ladder's mid tier, so for 38/64 tasks it grades output from its own model family. Evaluators favour their own generations, which biases the measurement toward OVERSTATING routed savings -- the reported cost inversion is therefore conservative. Also discloses seven generation calls retried at a raised token budget after returning empty output.
Adds the remaining three quarters of the programme alongside the empirical
refutation already on this branch, and a research/README.md that indexes all four
layers and leads with what the measurement actually showed.
New under research/cognitive-substrate/:
- the Theory -> Evidence -> Build-Map whitepaper (48pp, HTML + PDF)
- EXECUTIVE_SUMMARY.md, carrying a status banner: its prototype numbers are
refuted by research/empirical-refutation/ and the banner says so up front
- literature/ gap map + 32 graded references behind each faculty claim
- evidence/ twelve industry statistics independently re-grounded and graded
confirmed/vendor-reported/unverifiable (three widely-repeated
statistics were caught as misattributed and dropped), plus the
ecosystem map of what the 2026 stack already solves
- quranic-lens/ the fourteen-mapping ethical-epistemic design lens, with its
framing caveat: it names which safeguards are obligatory, and is
never offered as technical authority for an engineering claim
- sources/ the primary documents the evidence layer was graded against
- figures/ architecture schematics and prototype evaluations
Also corrects eval_precision_recall.png, whose panel title claimed the oracle had
the highest F1 while the same panel plotted grep higher, and marks the figure
superseded.
No source code changed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Uploads the full research programme behind forgekit as a single
research/tree with a masterindex, and adds the pre-registered empirical evaluation that refuted two of the programme's own
headline claims.
That refutation is the contribution. The later work corrects the earlier work, and the corrections
are the most useful part — so
research/README.mdleads with what was actually measured, not withwhat was originally claimed.
What the measurement showed
91% of files got exactly zero recall, and a
grepbaseline with no dependency graph beat thegraph-based oracle by ~10× on F1. The cost result is the sharpest: routing genuinely saves 59.5% on
first attempts, but almost none of that cheaper output is correct, and once escalation retries are
counted the pipeline costs more than always using the premium tier. The saving did not shrink —
it inverted.
Diagnosis, then repair
Classifying all 18,221 false negatives showed 96.8% were not a ground-truth ceiling: the path
existed in the graph the tool already builds. 94.7% were sibling pairs reachable only by a
mixed-direction path that reverse-only traversal cannot take, plus a construction defect that broke
one of the two standard Python package layouts.
Repairing both, with all eight new parameters fixed on six repositories and frozen before the three
held-out repositories were touched even once:
Beats the baseline for the first time. The margin is narrow, and precision generalised distinctly
worse than recall (−0.141 vs −0.021) — visible only because the split was declared in advance.
What is in this PR
Honesty notes a reviewer should check
EXECUTIVE_SUMMARY.mdcarries a status banner saying its own prototype numbers are refuted,with a table of what replaced them. Uploading it unmarked would have been misleading.
and dropped rather than repeated.
two concurrent preprints derive a strictly more general form weeks earlier. What survives is that
both are simulation-only; this work has real repositories.
such a theorem guarantees completeness only relative to the relation the closure runs over.
before correction — including two cases of reporting the more favourable of several available
figures, which is exactly the failure the paper indicts.
its own model family. That biases toward overstating savings, which makes the reported inversion
conservative.
internal consistency and the append-only amendment trail, but must take the ordering on trust.
eval_precision_recall.png) had a panel title claiming the oracle held the highest F1while the same panel plotted
grephigher. Corrected and marked superseded.Review notes
replacing
research/python-prototypes/impact_oracle/, so that swap stays your decision.