It sounds like this“Green jelly beans came out p = 0.04, so there's only a 5% chance it's a fluke, however many colours they tried.”
Judging one hit among many tests as if it were the only test
Why it feels right
Each test, taken alone, is honest: with no real effect it has only a 5% chance of a false alarm, and that figure is printed on every result. So when green jelly beans come back at p = 0.04, 'there's only a 5% chance it's a fluke' feels like the fair reading. What it leaves out is how many colours you tried before you found the green.
Why it’s wrong
Each test has a 5% chance of a false hit, so across 20 tests of things that do nothing, the chance that at least one hits is 1 - 0.95^20, about 64%. A single hit among many tries is what no effect looks like.
A worked example
Twenty colours of jelly bean are tested for a link to acne. None of them does anything. Each test still has a 5% chance of looking significant by luck, so each one stays clear 95% of the time. All twenty stay clear with probability 0.95 multiplied by itself twenty times, which is 0.36. The chance that at least one colour comes out 'significant' is 1 minus 0.36, or 64%.
So in nearly two out of three such studies, some colour lights up. The newspaper never hears about the other nineteen. It hears 'Green jelly beans linked to acne (p = 0.04)'.
This is how a lone hit among many tries is exactly what no effect looks like. The more things you check, the more certain it is that something will cross the line.
How to catch yourself
- You find one surprising result in a long list of things measured and read it on its own.
- You cannot find out how many outcomes, subgroups or variations the researchers looked at.
- A study 'found a link' only in one subgroup, one outcome or one time window.
What to hold instead
Ask how often at least one of the checks would come out significant if nothing were going on. That chance grows with every check: 64% with 20. Either tighten the bar to account for the number of checks (dividing 0.05 by the number of tests is the simplest correction), or treat a lone hit as a lead to confirm in a fresh study rather than a finding.
Where you’ll meet it
- Nutrition studies that measure dozens of foods against dozens of outcomes.
- Clinical trials that report only the subgroup in which the drug worked.
- Product teams reading twenty metrics on an A/B test and acting on the one that moved.
Questions people ask
- What is p-hacking?
- Trying many analyses, outcomes or subgroups until one gives p < 0.05, and reporting that one as if it were the only test. It does not need bad intent: it can happen just by following the data wherever it seems to lead.
- How do you correct for running many tests?
- The simplest way is the Bonferroni correction: divide your cut-off by the number of tests. With 20 tests you would need p < 0.0025 for any one of them to count. Better still, say in advance what you will test, and confirm any surprise on new data.