It sounds like this“p = 0.03 means there's only a 3% chance the drug doesn't work, so it's 97% likely to be real.”
Reading p as the chance the result is a fluke
Why it feels right
It is how almost every headline puts it, and it is the question you actually want answered: how likely is it that this result is just luck? A p-value sounds like exactly that number, and 'only a 3% chance it's a fluke' is a comforting sentence to read. The catch is that a p-value answers the reverse question. It starts by assuming the drug does nothing and asks how surprising your data would be in that world.
Why it’s wrong
The p-value assumes the drug does nothing and asks how often results this strong would turn up anyway. That is the chance of the data given no effect, not the chance of no effect given the data, and the two can be very different.
A worked example
Someone claims to be psychic. You test them: ten coin flips hidden from view, and they call 9 of the 10 correctly. If they were only guessing, the chance of 9 or more right is 11 in 1,024, about 1.1%. That is the p-value: p = 0.011.
Does it mean there is a 98.9% chance they are psychic? Of course not. Test 10,000 people who are all just guessing and about 107 of them will call 9 or more. If real psychics are as rare as you think they are, almost everyone who passes is a lucky guesser. The p-value never asked how plausible psychic powers were in the first place, so it cannot tell you how likely this person is to be one.
The same gap sits under every p-value. 'Data this strong are rare if nothing is going on' is one statement. 'Something is probably going on' is another, and it needs more than the data to get there.
How to catch yourself
- You describe a p-value as 'the chance it's a fluke' or 'the chance it was due to chance'.
- You turn p into confidence by subtracting it from 1, so that p = 0.03 becomes 'we're 97% sure'.
- You judge a result by its p-value alone, without asking how believable the idea was before the study.
What to hold instead
A p-value is the chance of results at least this extreme in a world where there is no effect. It is a statement about the data in that imagined world, not about the world you are in. To get from there to 'how likely is this real?', you also need to know how plausible the idea was beforehand and how big the effect is. A small p is a reason to look closer, not a verdict.
Where you’ll meet it
- Headlines that say a study 'proves' something at p < 0.05.
- A/B testing dashboards that show '95% confidence', which is usually just 1 minus the p-value.
- Medical papers, where a lone surprising result is quoted as established.
Questions people ask
- Is p = 0.03 a 3% chance the result is due to chance?
- No. It is the chance of seeing results this strong if there were no real effect. The chance that the result is a fluke, given the data, is a different number, and it depends on how likely the effect was to be real before you ran the study.
- So how likely is a significant result to be real?
- The p-value cannot say on its own. It depends on how many of the ideas being tested are real to begin with, and on how well the study can pick up a real effect. When real effects are rare, a large share of significant results are flukes.