It sounds like this“p = 0.0001 is way below 0.05, so the effect must be huge.”
Reading a tiny p as a big effect
Why it feels right
'Highly significant' sounds like 'highly important', and a p-value of 0.0001 looks a lot more impressive than 0.04. We are used to scores where smaller or bigger means stronger, so it seems natural to read a tiny p as a strong effect. But p measures how clearly the data stand out from 'no effect', and how clearly you can see something depends as much on how many people you measured as on how big it is.
Why it’s wrong
p shrinks as the study grows, even when the effect stays the same size. With 200,000 people a 0.1 kg difference gets p below 0.0001; it is real and too small to matter.
A worked example
A diet pill makes people 0.1 kg lighter on average. People's weights vary a lot, by about 5 kg either way, so in a small study that difference vanishes in the noise. With 50 people in each group the p-value is about 0.92: nothing to see.
Now run the same pill, with exactly the same 0.1 kg effect, on 100,000 people in each group. The noise averages out and the p-value drops below 0.00001. It is the same pill and the same tiny effect, and the result is now 'highly significant'.
Meanwhile a different pill that takes off 3 kg, tried on only 50 people per group, gets p = 0.003. The pill with the trivial effect has the far smaller p-value. If you ranked them by p, you would pick the wrong one.
How to catch yourself
- You call a result 'highly significant' and mean 'large'.
- You compare two studies by p-value to decide which effect is bigger.
- You read the p-value in a paper but skip the sentence that gives the size of the difference.
What to hold instead
p-values blend two things: how big the effect is and how much data there is. Read the size first, in kilos or minutes or points, and ask whether a person would notice it or care. Then use the p-value, or better the range around the estimate, to see how well it was measured. A tiny p with a tiny effect is a well-measured nothing.
Where you’ll meet it
- Nutrition and supplement news, where huge samples can make trivial differences 'significant'.
- Tech companies' A/B tests with millions of users, where a 0.05% change in clicks can be significant and worthless.
- Opinion polls and surveys of very large panels, where every small gap between groups looks real.
Questions people ask
- Can a result be statistically significant but not important?
- Yes, and with large studies it is common. Statistical significance says a difference is unlikely to be chance. It says nothing about whether the difference is big enough to matter, which you judge from the size of the effect.
- What should I look at instead of the p-value?
- The effect size, in units you understand, and the confidence interval around it. Together they say how big the effect probably is and how precisely it was measured.