It sounds like this“p = 0.06, so the diet doesn't work. The study proved it.”
Reading 'not significant' as 'proven to do nothing'
Why it feels right
We are taught to treat 'significant' as a verdict, so a test that comes back 'not significant' sounds like a test that looked and found nothing. And it is easy to believe that if there were a real effect, the study would have caught it. But a study can only catch what its size lets it see, and a small study sees very little.
Why it’s wrong
p = 0.06 means results like these would turn up fairly often even with no effect, so 'no effect' can't be ruled out. A study of 40 people that found 3 kg can't rule out a 6 kg effect either. It was too small to tell.
A worked example
Forty people try a new diet: 20 follow it and 20 do not. The dieters lose 3 kg more on average. Weight loss varies about 5 kg from person to person, which is a typical spread. The p-value comes out at about 0.06, just short of 0.05, and the newspaper writes: 'Diet doesn't work, study finds.'
Look at what the data can rule out. The plausible range for the true difference runs from about 0.1 kg of weight gain to about 6 kg of weight loss. The study cannot rule out no effect, and it cannot rule out an effect twice the size it measured. It was too small to tell.
Run the same diet on 400 people and the same 3 kg gap would give a p-value around 0.000000002. Same diet, same average result. The difference is how much the study could see.
How to catch yourself
- You read 'no significant difference' as 'no difference'.
- A headline says a treatment 'doesn't work' on the strength of one small study.
- You have never checked how many people were in a study that found nothing.
What to hold instead
'Not significant' means the study could not tell. Ask what range of effects the data cannot rule out, the confidence interval. If it runs from nothing to a big effect, the honest conclusion is 'too small to say'. Only a narrow range around zero shows that an effect is small or absent.
Where you’ll meet it
- Medical news that reports 'no evidence of benefit' when the trial was underpowered.
- Supplements and diets dismissed after one small study.
- Policy pilots that are cancelled because the first results were not significant.
Questions people ask
- Does p > 0.05 mean a treatment doesn't work?
- No. It means the study did not rule out 'no effect'. A treatment can work and still fail to reach p < 0.05, especially in a small study. The confidence interval shows what effects the data are still compatible with.
- What is the difference between 'no evidence of an effect' and 'evidence of no effect'?
- The first says the study did not find one, which could be because it was too small. The second says the study was large enough to rule out anything but a tiny effect. Many headlines blur the two.