What a p-Value Actually Means (and the Ways People Misread It)
A p-value isn’t the chance your result is wrong, or that it’s right. What it measures, with coin flips and a lady tasting tea, and why p < 0.05 isn’t magic.
Suppose a friend hands you a coin and you want to know whether it’s fair. You flip it 20 times and get 16 heads.
That looks like a lot. But fair coins do sometimes give lopsided results, so it isn’t proof. What you want to know is how surprising 16 heads would be if the coin were fair. So you ask a very specific question:
If this coin were fair, how often would I see a result at least this extreme?
The answer, worked out from the maths of coin flips, is about 1.2% of the time, counting 16 or more heads or 16 or more tails. That number is the p-value: 0.012.
Try it
How often does a fair coin look this lopsided?
A thousand experiments, each 20 flips of a fair coin. Move the slider to the result you got.
With a fair coin, 16 or more heads (or 16 or more tails) came up in 0.0% of these experiments. That’s the p-value for your result: 1.2% exactly. It says how surprising your result would be if the coin were fair, not the chance that it is.
A lady, a cup of tea and the birth of the p-value
The idea is almost a century old. In the 1920s, at an afternoon tea at the Rothamsted agricultural research station, a scientist named Muriel Bristol told the statistician Ronald Fisher that she could taste whether milk had been poured into a cup before or after the tea. Fisher, being Fisher, turned it into an experiment: eight cups, four made each way, shuffled out of her sight.
He asked how likely she’d be to name all eight correctly by pure guessing. There are 70 ways to choose which four cups had milk first, so the chance of guessing right by luck is 1 in 70, about 1.4%. If she got them all right, it would be hard to put down to luck. That’s a p-value: the chance of a result this good if she were only guessing. (She’s said to have got them all right.)
Fisher went on to write that a p-value of about 0.05, or one in twenty, was a convenient place to call a result significant. “Convenient” is the word to hold on to. It was a rule of thumb for deciding what was worth a second look, and over the next hundred years it hardened into something much more rigid than he meant.
The definition
A p-value is:
The probability of getting data at least as extreme as yours, assuming there is no real effect.
The assumption of “no real effect” is called the null hypothesis. For the coin it’s “the coin is fair”. For the lady it’s “she’s guessing”. For a drug trial it’s “the drug does nothing”.
A small p-value means your data would be surprising if the null were true. That’s a reason to doubt the null. It isn’t proof of anything, and it isn’t a probability about the null. The whole calculation lives inside the imaginary world where nothing is going on.
The misreadings
1. “There’s a 1.2% chance the coin is fair”
No. The p-value was calculated by assuming the coin is fair, so it can’t then tell you the probability that it is. To answer that, you’d need to know how likely a trick coin was in the first place. That’s the same base-rate problem as in base rate neglect: a surprising result from a coin you’ve every reason to think is fair may still be a fluke. There’s a longer look at this one, with a worked example, in reading p as the chance the result is a fluke.
2. “A small p means a big effect”
No. A p-value mixes two things: how big the effect is and how much data you have. With enough people, a trivially small difference gets a tiny p-value. A study of 200,000 people can find that a pill makes people 0.1 kg lighter with p below 0.0001. It’s real and it’s useless. See reading a tiny p as a big effect.
3. “A big p means no effect”
No. It means the study didn’t find evidence of one. A small study can miss a real difference easily. A study of 40 people finding that a diet works about as well as no diet isn’t proof that the diet fails: it may simply have been too small to tell. See reading ‘not significant’ as proven to do nothing.
Try it
How many findings held up?
In 2015, a large team tried to repeat 100 published psychology experiments. In 97 of the originals, the result had been statistically significant. How many of the repeats got a significant result too?
Drag the dot to your guess, then lock it in. Nothing is revealed until you do.
4. “95% of significant results are true”
No. The 5% is the share of duds that sneak through, not the share of passes that are duds. If most of the ideas a field tests are wrong, a big share of its significant results will be flukes. This is why the replication numbers are so sobering, and it’s the subject of thinking a significant result is 95% sure to be real.
The multiplying problem
Here’s the quietest way p-values go wrong. Each test at the 5% level has a 5% chance of a false alarm when nothing’s going on. Run twenty such tests and the chance that at least one of them lights up is not 5%, it’s about 64%.
The webcomic xkcd made this famous with a team of scientists testing 20 colours of jelly bean for a link to acne. Nineteen did nothing. The green ones came out at p < 0.05, and the headline read “Green jelly beans linked to acne!” It’s funny because it’s the way real research works when the counting is hidden: what gets reported is the hit, and the nineteen misses vanish.
Try it
Twenty things that do nothing
You test lots of things that really have no effect, each at the usual p < 0.05 cut-off. How likely is it that at least one comes out “significant” by luck?
Test 20 things that do nothing and there’s a 64% chance of at least one “significant” result, and you’d expect about 1.0 of them. One hit among many tries is what no effect looks like, and it’s the one that gets written up.
This is what people mean by p-hacking: trying many outcomes, subgroups or analyses, and stopping when one gives p < 0.05. It doesn’t need bad intent. A researcher following the data wherever it seems to lead can do it by accident. The defence is to say in advance what you’ll test, and to be suspicious of a single hit among many tries.
When a significant result wasn’t
In 2011, the psychologist Daryl Bem published nine experiments in one of the top journals in the field, eight of which showed statistically significant evidence for precognition: people seemed to respond to things before they happened. Every one followed the standard rules. The result wasn’t that psychologists believed in ESP. It was that they asked how an unbelievable result could have passed their own checks, and the answer made them uncomfortable.
In 2015 the Open Science Collaboration tried to repeat 100 published psychology experiments. In 97 of the originals the result had been significant. In the repeats, only 36 were. That doesn’t mean the other studies were fraudulent. Many were probably flukes that passed the p < 0.05 bar and were then published, because a significant result is much more likely to be published than a non-significant one.
What to look at instead
A p-value isn’t useless. It’s one number that answers one question. The trouble is treating it as the whole analysis. In 2016, the American Statistical Association took the unusual step of issuing a formal statement on its misuse, which includes the warning that a p-value says nothing about the size of an effect or the importance of a result.
What to ask for alongside it:
- The effect size, in units you can picture. How many kilos, minutes or points?
- The confidence interval: the range of effects the data are compatible with. If it runs from nothing to huge, the study didn’t tell you much, whatever the p.
- How many things were tried. Was this the one test, or the best of twenty?
- Whether it’s been repeated. A result that two different teams find is worth much more than one that a single team did.
Physicists, incidentally, don’t use 0.05. The discovery of the Higgs boson was announced at “five sigma”, a p-value of about 0.0000003. A field that can afford to demand more evidence tends to.
The takeaway
A p-value is a measure of surprise, calculated in a world where nothing is going on. Used that way, it’s a useful warning light: this result would be unusual if there were no effect. Used as a verdict on whether something is true, it misleads, and it has done so for a century. If you’d like to see where your own instincts stand, the free statistics test puts the p-value alongside correlation, averages and polls.
Questions people ask
- What is a p-value in simple terms?
- It’s how surprising your data would be if there were no real effect. If a drug did nothing, how often would a study give results at least this strong, just by chance? A small answer means the data would be unusual in a no-effect world, which is a reason to doubt that world.
- What does p < 0.05 mean?
- That data at least this extreme would turn up less than 5% of the time if there were no effect. Researchers conventionally call that “statistically significant”, but the 5% line is a convention from the 1920s. It isn’t a rule of nature and shouldn’t decide whether something is true or important.
- Is the p-value the probability that the null hypothesis is true?
- No. The p-value is calculated assuming the null hypothesis is true, so it can’t also tell you how likely that assumption is. To get that you need extra information, such as how plausible the idea was before the study, which is the territory of Bayesian reasoning.
- Does a large p-value mean there is no effect?
- No. It means the study didn’t find evidence against no effect. A small study can easily miss a real one. The confidence interval, which shows the range of effects the data are compatible with, is more informative than the p-value alone.
- What is p-hacking?
- Trying different analyses, outcomes or subsets until one gives p < 0.05, then reporting only that one. Because chance throws up small p-values regularly, enough tries will find one even when nothing’s going on. Pre-registering what you’ll test in advance is the usual defence.
Sources and further reading
- Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver and Boyd.
- Fisher, R. A. (1935). The Design of Experiments. Oliver and Boyd.
- Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133.
- Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716.
- Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100(3), 407–425.
- Wasserstein, R. L., Schirm, A. L. and Lazar, N. A. (2019). Moving to a world beyond “p < 0.05”. The American Statistician, 73(sup1), 1–19.
