It sounds like this“p < 0.05 means a 5% error rate, so 95% of significant results are real.”
Thinking a significant result is 95% sure to be real
Why it feels right
'Significant at the 5% level' sounds like 'a 5% chance I'm wrong', and so 95% sure I'm right. The 5% is real, but it is the share of duds that sneak through, not the share of results that are duds. Those are different groups of people, and the second depends on something the p-value never sees: how many of the ideas being tested were worth anything.
Why it’s wrong
5% is the share of duds that pass, not the share of passes that are duds. If 900 of 1,000 ideas are duds, 45 pass by chance beside 80 real hits, so 36% of the significant results are flukes.
A worked example
A lab tests 1,000 ideas. In a good field, say 100 are real and 900 do nothing. The studies catch 80% of the real ones: 80 pass. And 5% of the 900 duds pass by chance: 45. That makes 125 significant results, of which 45 are duds. So 36% of the 'significant' findings are flukes, not 5%.
Make real effects rarer, as in a speculative field where 1 idea in 100 is real. Then 10 are real and 8 of them pass, while 990 duds put 49 or 50 flukes through. About 86% of the significant results are flukes.
Make them common, with half the ideas real, and the picture flips: 400 real ones pass beside 25 flukes, so only about 6% are duds. The p < 0.05 bar is identical in all three. What changed is how many of the ideas were true to begin with.
How to catch yourself
- You treat 'p < 0.05' as 'a 95% chance this is right'.
- You do not ask how surprising the claim was before the study.
- You trust a lone surprising result in a field where most ideas fail.
What to hold instead
The 5% is the share of duds that pass. The share of passes that are duds depends on how many of the ideas tested were duds. The rarer real effects are, the more of the significant results are false. This is why surprising claims need stronger evidence than plausible ones, and why a result that has been replicated is worth far more than one that has not.
Where you’ll meet it
- The replication crisis: why many published findings did not hold up when repeated.
- Drug discovery and genetics, where thousands of candidates are screened for a few real hits.
- Any screening test for a rare condition, which has the same shape of problem.
Questions people ask
- What is the false discovery rate?
- The share of significant results that are really duds. It is not the same as the 5% significance level, and it depends on how common real effects are among the ideas tested and on how well studies detect them.
- Why do so many published studies fail to replicate?
- One reason is just this arithmetic. When many tested ideas are wrong, a good share of the ones that reach p < 0.05 are flukes, and flukes do not repeat. Running many tests and reporting the hits makes it worse.