A small, pre-registered computer experiment by Ron (RonBuildsAI), built and run with an AI assistant. Plain Python, nothing to install, about 10 seconds to run.
When you test whether something works, a fair test still says "it worked" about 5 times in 100 by luck alone. That is the accepted false-alarm rate (p below 0.05).
But if you look at the data first and then choose how to test it (which measure, which subgroup, whether to stop early or keep going, whether to drop odd data points), every choice is another lottery ticket. How often do you then "find" something that is not there?
This program measures that. It builds 10,000 simulated studies where we KNOW nothing is true (two groups of 20, drawn from the same pool), and two analysts judge every one:
- Arm F (frozen): writes the test down before seeing any data. One measure, one sample, one test.
- Arm L (look first): sees the data, then tries up to 120 ways to slice it, and reports a win if any one of them works.
Every "win" in this world is a fake win, so they can be counted exactly.
| World (10,000 studies each) | Arm F, test written first | Arm L, look first (120 forks) |
|---|---|---|
| Primary: measures correlated 0.5 | 497 = 5.0% (99% interval 4.4 to 5.6) | 7,712 = 77.1% (99% interval 76.0 to 78.2) |
| Report only: correlation 0.2 | 5.3% | 82.2% |
| Report only: correlation 0.8 | 5.0% | 65.9% |
Writing the test first held fake wins at the promised 5%. Looking first produced them 77% of the time, 15.5 times as often. The bar written before the run (arm L at least double arm F, with separate intervals) passed.
Where the fake wins come from (primary world, arm F's test plus one freedom at a time): peeking early at 10, 15 and 20 per group gives 9.5%; choosing among the outcomes 13.3%; choosing a subgroup 17.5%; dropping odd points or not 10.5%. Each one alone roughly doubles or triples the 5%. Together they compound to 77%.
The other side of the price: with a real effect planted (d = 0.8), arm F found it 68.8% of the time (its computed power) and arm L 98.9%. Freedom finds real effects more often too, which is why it tempts. Its cost is paid on the null.
All of the above was decided before the run in the frozen pre-registration (PREREG_PUBLIC.md, see the note at its top) and computed by the program after an instrument gate passed: its math self-test, two planted mistakes it had to catch, one known answer, and an independent second implementation of arm L.
Computed afterwards from the same saved rows (real effects at d = 0.8, 20 per group). These are directions to test, not findings:
| Method | Fake wins | Real effects found |
|---|---|---|
| Look first alone | 77.12% | 98.87% |
| Test written first, bar 0.05 | 4.97% | 68.77% |
| Test written first, bar 0.01 | 0.88% | 43.23% |
| Explore freely, then confirm on fresh data with a test written first (0.05) | 3.94% | 67.60% |
| Two independent tests written first, both at 0.05 | 0.38% | 47.26% |
The pattern: at a fixed sample size there is a dial, not a free lunch. Every step that cuts fake wins also misses more real effects. Exploring freely costs almost nothing if the find must then be confirmed on fresh data under a test written first. Pushing both errors down at once takes more data per test.
This agrees in direction with two well-known papers. It is a replication of their argument on our own instrument, not a new discovery:
- Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. Their simulation found that four common choices (two outcome measures, adding more subjects after peeking, controlling for gender, dropping one of three conditions) together raised the false-positive rate from 5% to about 61%. We used 120 forks and got 77%.
- Gelman, A., and Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no "fishing expedition" or "p-hacking" and the research hypothesis was posited ahead of time. Working paper, Department of Statistics, Columbia University.
You need Python 3. Nothing to install (standard library only). In this folder:
python rule_test.py selftest
python rule_test.py smoke
python rule_test.py real --prereg PREREG_PUBLIC.md --out results_real_mine
selftestchecks the math only. Seconds.smokeis a small run on practice seeds. It prints the instrument checks and, on purpose, no arm L rate.realis the full run on the frozen seeds. It first checks that the pre-registration is frozen, has its predictions filled, and names this program's exact sha256; then runs for about 10 seconds (7 s MEASURED on our Windows desktop) with up to 4 worker processes; then runs the instrument gate; and only if the gate passes, prints the results and writes theresults_real_minefolder.
What you should see: GATE PASSED, and in the primary world (W1_NULL) arm F at 497 of 10,000 and arm L at 7,712 of 10,000. Compare your results_real_mine/summary.json with our_results/summary.json; the per-study row hashes should match too.
One real run per copy, on purpose: the program refuses a second real run once a results_real_* folder sits next to it. That is the "one read per pre-registration" rule built into the code. To run again, use a fresh copy.
On Windows, if python is not found, try py.
If the real run says it is "refused" because the pre-registration "does not name this code's sha256", your copy of rule_test.py was changed on the way to you (most often line endings). Download the ZIP from GitHub, or clone again, and do not open and re-save the file in an editor first.
- Reproduce: run it as is. You should get exactly our numbers. This tests that the program behaves the same on your computer.
- Re-implement (the strongest check): read sections 3 to 5 of
PREREG_PUBLIC.mdand write your own version without openingrule_test.py. If you land near 5% and 77%, our code is not hiding a mistake that only our code has. - Replicate on new seeds (advanced): change the seeds in the program. The program will then refuse the real run, because the frozen file names the old code's sha256. That is the method working: write your own frozen copy that names your new code's hash (the refusal message prints it), fill in your own prediction first, and then run.
Whichever you pick, write down your guess before you run: out of 100 studies where nothing is true, how many fake wins for each analyst?
- It is one simulated kind of study (Gaussian data, 20 per group, three correlated measures plus their average, two coin-flip subgroups, three looks). It is not a measure of any real analyst, any real study, or any of Ron's own tests.
- Arm L uses every fork on its list and none off it. A real analyst uses fewer of these (rate lower) and has others (rate higher).
- Exact reproduction depends on the platform's math library for random numbers. MEASURED: the full real run reproduced byte for byte, every per-study row in all five worlds, on a Linux desktop (Python 3.12.3, the original run) and a Windows desktop (Python 3.14.5, this public copy, 2026-10-11). Other machines are untested, which is one reason we would like you to run it.
| File | What it is | sha256 |
|---|---|---|
rule_test.py |
the program, exactly as frozen | d759718a66a0e11edbed20aa8a7a70c37c370d3b1bc3b9bdf72cf20cdd6d13b6 |
PREREG_PUBLIC.md |
the frozen pre-registration, public copy with every removal marked | ee8b6d023c904c6202537d750d936bb0fc4b92d0f79b94a58f430c971444514a (the original frozen file's is 5e63598c260897e67fbb261f21cc21ba9d657447e4b9a9a82192d7fac7caeb40) |
.gitattributes |
tells git never to change line endings, so the program's sha256 survives a download | |
our_results/summary.json |
the counts from our one real run | f0386f5a0b8e0f156e712e7ca711f5b970b3a271c86947a9fd1fded31145a695 |
our_results/rows_*.csv |
every simulated study's flags, one row each | hashes inside summary.json |
Drafted and fact-checked with an AI assistant from my own records, measurements and decisions. I read and edited every sentence and I am responsible for its contents.