An A/B analysis of Cookie Cats, a mobile puzzle game: 90,189 players were randomised on install, and the experiment moved the game's first progression gate from level 30 to level 40. A gate blocks progress until the player waits or pays, so moving it later should mean more play before the first wall — the question is whether that helps or hurts retention.
Answer: it hurts. Keep the gate at level 30.
The statistics are written out by hand (scipy for distributions only, no statsmodels),
because the arithmetic is the part worth being able to defend.
Start here: notebooks/01_gate_experiment_walkthrough.ipynb
— the analysis as a narrative, executed with outputs embedded, calling the same tested
package rather than reimplementing it.
| Metric | gate_30 | gate_40 | Difference | 95% CI | p | Holm |
|---|---|---|---|---|---|---|
| 1-day retention | 0.4482 | 0.4423 | −0.59 pp (−1.3%) | [−1.24, +0.06] pp | 0.0744 | not rejected |
| 7-day retention | 0.1902 | 0.1820 | −0.82 pp (−4.3%) | [−1.33, −0.31] pp | 0.0016 | rejected |
Moving the gate to level 40 costs 0.82 percentage points of 7-day retention, and the result survives correction for testing two metrics. Retention compounds — a permanently smaller returning player base is not recovered later — so the recommendation is to leave the gate where it is.
Anyone can call proportions_ztest. The six things below are what the analysis is for.
Before any metric: Sample Ratio Mismatch. Randomisation is the assumption every other number rests on, so it gets tested first.
gate_30: 44,700 gate_40: 45,489
observed control share 0.49563 (intended 0.5)
chi-square 6.902, p = 0.00861
This flags at α = 0.05. Most write-ups of this dataset never check, and the ones that do usually stop there and declare the experiment broken. Neither reaction is right.
An SRM check runs on every experiment, every day — at α = 0.05, one healthy experiment in
twenty trips the alarm. Industry practice is to alert at p < 0.001, and this split
does not reach that. So the honest reading is: the imbalance is worth a look at the
assignment service, not worth discarding the experiment. Both verdicts are reported
(stats.srm_check) because which threshold applies is a
policy decision, not a statistical one.
1-day retention came back at p = 0.074. The tempting sentence is "the gate has no effect on day-1 retention". The data does not support it:
retention_1: MDE at 80% power = 0.932 pp observed −0.591 pp exceeds MDE: False
retention_7: MDE at 80% power = 0.735 pp observed −0.820 pp exceeds MDE: True
The experiment could only reliably detect a day-1 effect of 0.93 pp or larger, and the observed effect is smaller than that. The correct statement is "any day-1 effect below 0.93 pp would have been invisible to us" — which is a different claim, and an honest one. Note how neatly this explains both results: day-7 cleared its MDE and came back significant, day-1 did not and did not.
The z-statistic uses the pooled proportion — under the null the two arms share one true rate, so all the data should estimate it. The confidence interval uses the unpooled error — an interval describes the effect without assuming it is zero, and assuming the null while estimating the effect size would be circular.
Using one SE for both is the most common bug in hand-rolled A/B code. It rarely changes the verdict, which is exactly why it survives.
A percentile bootstrap is run as a second opinion, since it leans on no normal approximation at all. It agrees to within 0.02 pp:
Rounds played is not a candidate for a t-test. The median is 16, the mean is 51.9, and one player logged 49,854 rounds — a single observation that moves the mean more than the treatment does.
So the guardrail uses Mann-Whitney U (p = 0.0502, P(treatment > control) = 0.496). What it tests is stochastic dominance, not a difference in means — a distinction that matters when writing the sentence, because a significant U does not license "players played more on average".
A gate at level 30 cannot affect a player who never reaches level 30, and 4.4% of players never played a single round. So the measured effect is diluted, and the obvious fix is to filter to players who actually got that far.
That filter is invalid, and the project shows why with numbers rather than assertion.
sum_gamerounds is measured after randomisation and is itself affected by the treatment
— a gate is precisely a thing that stops people playing. Filtering on it therefore
conditions on a post-treatment variable and breaks the randomisation:
filtering to players with >= 30 rounds: diff −0.874 pp, p = 0.1080
but the filter itself is treated: P(passes filter) 0.3726 → 0.3652, p = 0.02114
The second line is the proof: the two "filtered" populations are not comparable, because treatment changed who survives the filter. This is the same structure as survivorship bias, and it reliably produces a confident wrong answer.
The legitimate versions: segment on a variable measured before assignment, or declare the segment in advance and randomise within it. Neither is available in this dataset, and saying so is better than producing the number anyway.
The result in §1–§5 is a fixed-horizon analysis: valid because the sample size was fixed in advance and the data was inspected once, at the end. That is how the arithmetic works, and it is not how experiments are run — the dashboard is live and somebody opens it on Tuesday.
What peeking costs, measured on this experiment. Simulated A/A tests at this experiment's own base rate (19.0%) and sample size (44,700 per arm), stopping at the first look that clears α = 0.05:
| Times the dashboard is checked | 1 | 2 | 5 | 10 | 20 |
|---|---|---|---|---|---|
| False positive rate | 4.7 % | 8.2 % | 14.4 % | 19.5 % | 25.2 % |
The single-look column is the control: it comes back at 4.7%, so the simulator is measuring the effect rather than inventing it. The shape is the lesson — the damage is done by the first few extra looks. Checking twice already costs most of what checking ten times does, so "we only glanced at it once in the middle" is not the defence it sounds like.
What would have been legitimate. An asymptotic confidence sequence (Waudby-Smith, Arun, Howard & Ramdas, 2021) — an interval valid at every sample size at once, so stopping at the first crossing is allowed. Replayed over the real experiment:
retention_7 fixed-horizon CI first clears zero at n = 14,327
always-valid CI first clears zero at n = 36,346 (2.5x later)
always-valid CI at the full sample: [-1.58, +0.01] pp
width 1.55x the fixed-horizon interval = 2.4x the users
retention_1 never clears zero - consistent with p = 0.074 and its MDE
Two things are worth saying plainly about that. First, the price is explicit: the licence to stop whenever you like costs about 2.4× the sample, which here is almost exactly the difference between a clear result and a marginal one. Second, the always-valid interval clears zero at 36k and then drifts back over it by the full sample. That is not a contradiction — the guarantee is coverage at every time, and stopping at the first crossing is a legitimate use of it — but reporting only the crossing would imply the interval stays clear afterwards, which it need not.
The dataset carries no timestamps, so arrival order is simulated by shuffling. One shuffle would produce one stopping time and far too much confidence in it, so the stopping time is reported across 20 orderings: all 20 crossed at some point, between n = 12,919 and n = 44,700. The verdict at the full sample does not depend on the order.
pip install -e ".[dev]"python -m ab_test_analysisDownloads and caches the data on first run (2.7 MB), prints the full readout, writes
reports/results.json and regenerates every figure.
pytest -q76 tests, no network — the suite builds synthetic experiments with a known true effect and checks the pipeline recovers it: that a real effect is found with the right sign, that identical arms produce no winner, that MDE and required-sample-size invert each other, that the rank test barely moves when an outlier is injected while the mean moves 100×.
The sequential tests are the same idea applied to a guarantee rather than an estimate: under the null, the confidence sequence is checked to contain zero at every checkpoint in at least 95% of simulated experiments, while the ordinary interval read at the same checkpoints is checked to fail more often. A statistical guarantee that is asserted rather than measured is just a comment.
src/ab_test_analysis/
├── config.py the analysis plan, declared before the results are seen
├── data.py download, cache, and validation that fails loudly
├── stats.py SRM, z-test, bootstrap, Holm, MDE, Mann-Whitney — by hand
├── sequential.py the cost of peeking, and an interval that survives it
├── analysis.py the pipeline, in the order it must be run
├── plots.py five figures, every comparison with an interval
└── __main__.py the readout
config.py exists so the plan is readable before the results are: which metrics are
primary, what α is, which threshold alerts on SRM. Deciding those afterwards is how a null
result quietly becomes a positive one.
- One dataset, one experiment. No CUPED variance reduction and no heterogeneous treatment effects — both of which a production experimentation platform would have, and neither of which this dataset supports (CUPED needs a pre-period covariate; see the next bullet).
- The sequential analysis is retrospective. A confidence sequence earns its keep when it is declared up front and watched live. Here it is replayed over a finished experiment on a simulated arrival order, which is enough to price the guarantee and not enough to claim the experiment was run that way. Alpha-spending boundaries (Pocock, O'Brien-Fleming) are the alternative when the number of looks is known in advance — they buy back some of the 2.4× at the cost of fixing the schedule.
- No pre-treatment covariates. The dataset carries no attribute measured before assignment, which is why the dilution problem in §5 has no valid fix here rather than a better one.
- Retention is measured at fixed horizons, so players who install late in the collection window are censored; the dataset does not expose install dates to check it.
- The SRM cause is not diagnosable from this data. Flagging it and stating what would be checked next (assignment service logs, platform breakdown) is as far as the available data goes.