Numbers · 3 min read
What a p-Value Actually Means (and the Three Ways People Misread It)
A p-value isn't the chance your result is wrong, or the chance it's right. What it really measures, with a coin-flipping example, and why p < 0.05 isn't magic.
By the Cognosc team ·
The p-value is probably the most used and most misunderstood number in science. It appears in almost every study, it decides what gets published, and surveys find that many researchers can’t define it correctly. Here’s what it actually is.
Try it
How often does a fair coin look this lopsided?
A thousand experiments, each 20 flips of a fair coin. Move the slider to the result you got.
With a fair coin, 16 or more heads (or 16 or more tails) came up in 0.0% of these experiments. That’s the p-value for your result: 1.2% exactly. It says how surprising your result would be if the coin were fair, not the chance that it is.
Start with a coin
Suppose a friend hands you a coin and you want to know whether it’s fair. You flip it 20 times and get 16 heads.
That seems like a lot. But fair coins sometimes give lopsided results. So you ask a specific question:
If this coin were fair, how often would I see a result at least this extreme?
The answer, worked out from the maths of coin flips, is about 1.2% of the time (counting 16 or more heads, or 16 or more tails). That number is the p-value: 0.012.
The definition
A p-value is:
The probability of getting data at least as extreme as yours, assuming there is no real effect.
The “no real effect” assumption is called the null hypothesis. For the coin, it’s “the coin is fair”. For a drug trial, it’s “the drug does nothing”.
A small p-value means your data would be surprising if the null hypothesis were true. That’s a reason to doubt the null hypothesis. It isn’t proof of anything.
The three common misreadings
1. “There’s a 1.2% chance the coin is fair.”
No. The p-value was calculated assuming the coin is fair. It can’t then tell you the probability that it is. To know that, you’d need to know how likely a trick coin was to begin with, which is the base-rate problem from why a positive test often isn’t what it seems.
2. “There’s a 98.8% chance the effect is real.”
Same mistake, flipped around. A p-value of 0.012 doesn’t mean the effect is 98.8% likely to be real.
3. “The result is significant, so it’s important.”
“Statistically significant” means unlikely to be produced by chance alone, not big or important. With a huge sample, a tiny, meaningless effect can have a very small p-value. Always look at the size of the effect as well.
Why 0.05?
By convention, many fields call a result “statistically significant” if p is below 0.05, meaning data this extreme would turn up less than 1 time in 20 if there were no effect. The threshold dates back to the statistician Ronald Fisher in the 1920s, who suggested it as a rough guide, not a rule.
There’s nothing magic about it. A p-value of 0.049 isn’t meaningfully different from 0.051, and in 2016 the American Statistical Association issued a statement warning against treating the line as a verdict.
Why it goes wrong in practice
Testing lots of things. If you test 20 things that have no real effect, you’d expect about one of them to come out “significant” by chance. Studies that measure many outcomes and report the one that worked are a big source of false findings.
Stopping when it works. Checking the results repeatedly and stopping as soon as p dips below 0.05 inflates false positives.
Small studies. Small studies that happen to find big, significant effects are often overestimating them, and later, larger studies find smaller effects or none.
What to look for instead
- The effect size: how big is the difference, in real units?
- A confidence interval: the range of effect sizes consistent with the data.
- Replication: has anyone found the same thing again?
- The whole picture: how many things were tested, and was the analysis planned in advance?
Test yourself
The free statistics test includes the p-value question and nine other ideas that are easy to get backwards.