100 empty worlds
Generate a grid of "null worlds" — fake datasets with a genuinely random split and no real effect — and run the same significance test on every one. Then watch how many effect-free worlds still light up "significant" by chance alone. It's the fastest way to feel what a p-value actually promises, not just be told. Runs entirely in your browser (0 uploads, works offline).
How to use it
- Pick a significance threshold (0.01 / 0.05 / 0.10) and a sample size.
- Generate the worlds — each is pure noise, two samples from the same distribution.
- Count the highlighted worlds — the false positives — and re-roll to see it hold near the threshold.
What this clears up (the fundamentals)
- What a p-value is — p<0.05 doesn't mean "there's a real effect." It means "if there were no effect, a result this extreme would happen ~5% of the time." This shows you that 5%.
- False positives are guaranteed — run enough effect-free tests and some will be significant. At p<0.05, about 1 in 20, every time.
- The multiple-comparisons trap — test 100 things and ~5 will look "significant" with nothing real behind them. That's how p-hacking and dredged-up findings happen.
- Threshold as a dial — a stricter threshold (0.01) cuts false positives; a looser one (0.10) floods them in. The knob is a choice about how often you'll be fooled.
Where it's used
A Data literacy companion for anyone reading (or running) A/B tests and experiments — and a fast antidote to "but it was statistically significant!" It's a miatz build-lab concept playable; learn to build it yourself.
FAQ
What does a p-value actually mean?
It's the probability of seeing a result at least this extreme if there were no real effect. p<0.05 means "under the null, this happens ≤5% of the time" — not "95% chance the effect is real." This page makes that concrete by running the test on worlds you know have no effect.
Why do random, effect-free datasets come out "significant"?
Because "significant" is defined by chance. At p<0.05, roughly 5% of tests on pure noise will cross the line — not because there's an effect, but because random variation occasionally looks like one.
Is anything uploaded?
No. Every world is simulated in your browser with a seeded random generator, and nothing is transmitted or stored — turn off your Wi-Fi and it still works.
How does this relate to p-hacking?
If you test many hypotheses (or re-run until one "works"), you're sampling from these empty worlds until one turns green. This shows why a single significant result, absent pre-registration and correction, proves very little.
Limits
A teaching simulation using a two-sample z-test on normal data — real experiments use varied tests and distributions, but the lesson (false positives cluster at the threshold) is general.
Related
Part of the Demystify Playgrounds. Explore the rest from the Playgrounds home.
Bookmark this page (Ctrl+D, or ⌘D on Mac) — it works offline the next time you need it.