Peek at your own risk
Run an A/B test where there is no real difference — two identical arms — and watch what happens when you keep checking the results early. Each peek is another chance for random noise to cross the 5% significance line. Drag the number of peeks and see the true false-positive rate balloon far past the 5% you thought you were running. Simulated live in your browser (0 uploads, works offline).
How to use it
- Set how many times you peek at the results before the test is done.
- Compare the two bars — the 5% you assumed versus the actual false-positive rate across thousands of simulated no-effect tests.
- Read the X-ray: exactly how much stopping early on the first green result inflates your error rate, and why.
What this clears up (the fundamentals)
- Optional stopping is a multiple-comparison problem — a test fixed at 5% assumes one look at a pre-set sample size. Every extra peek is another independent shot at a false positive, so the errors compound.
- Noise wanders — with no real effect, the observed difference drifts up and down. Given enough looks, it will cross the threshold by chance; peeking lets you stop exactly when it does.
- The inflation is large and fast — even a handful of peeks can double or triple the 5% error rate. "We checked and it was significant" is not the same as "it's significant."
- The fixes are known — fix the sample size in advance, or use a sequential/alpha-spending method designed for continuous monitoring. Watching a dashboard and stopping when it's green is neither.
Where it's used
An experimentation-literacy drill — the fastest way to feel why "we saw significance on day 3 and shipped" is a trap, and why analytics tools warn against early stopping. It's a miatz build-lab concept playable; learn to build it yourself.
FAQ
What is an A/A test?
An A/B test where both arms are identical — there is genuinely no difference. Any "significant" result it produces is by definition a false positive, which makes it the perfect way to measure how often your process cries wolf.
Why does peeking inflate the false-positive rate?
Because each look is another opportunity for random fluctuation to exceed the significance threshold. The 5% guarantee applies to a single test at a fixed sample size; checking repeatedly and stopping at the first significant result is testing many times and keeping the luckiest look.
How is this simulated?
Thousands of A/A experiments run entirely in your browser: each accumulates samples for two identical arms and runs a z-test at every checkpoint, stopping if any crosses |z| > 1.96 (the 5% two-sided line). The false-positive rate is the fraction that stopped early.
Is anything uploaded?
No. Every simulated trial runs on your device — nothing is transmitted or stored. Turn off your Wi-Fi and it still works.
Limits
A teaching simulation with equal-variance normal data and evenly spaced looks — real tests have skewed metrics, unequal arms, and business-driven stopping. The direction and rough magnitude hold regardless: more peeks, more false positives. Use a proper sequential-testing tool for real decisions.
Related
Part of the Demystify Playgrounds. Explore the rest from the Playgrounds home.
Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.