cognoscJoin the beta

What does a p-value actually tell you?

A p-value isn't the chance your result is a fluke. Guess what p = 0.03 means, then see the imagined world it's worked out in, and why 0.05 misleads.

The steps5 · 24 min
  1. 1The chance of your data if nothing is going on5 min
  2. 2A tiny p doesn't mean a big effect5 min
  3. 3'Not significant' doesn't mean 'no effect'4 min
  4. 4Test enough things and one will pass5 min
  5. 5A significant result can still be a fluke5 min
Free, no card
What it takes apart

Sound like you?

The ideas people most often get wrong on the way, in their own words. Each step starts from one of them.

“p = 0.03 means there's only a 3% chance the drug doesn't work, so it's 97% likely to be real.”Taken apart in step 1Reading p as the chance the result is a fluke“p = 0.0001 is way below 0.05, so the effect must be huge.”Taken apart in step 2Reading a tiny p as a big effect“p = 0.06, so the diet doesn't work. The study proved it.”Taken apart in step 3Reading 'not significant' as 'proven to do nothing'“Green jelly beans came out p = 0.04, so there's only a 5% chance it's a fluke, however many colours they tried.”Taken apart in step 4Judging one hit among many tests as if it were the only test“p < 0.05 means a 5% error rate, so 95% of significant results are real.”Taken apart in step 5Thinking a significant result is 95% sure to be real
Statistics

More on statistics

Why use a database instead of just saving files?26 min · freeWhy is the Fourier transform so important?28 min · freeWhy doesn't the Moon fall onto Earth?27 min · freeWhy does salt make ice water colder?26 min · free
The short answerRather read it than work it out?Show it (it gives the steps away)

A p-value is worked out in an imagined world where there is no effect (the null hypothesis). It is the chance, in that world, of results at least as extreme as yours. It is not the chance that there is no effect, and not the chance your result is a fluke.

A p-value mixes how big the effect is with how much data there is. With enough people, a difference too small to matter gets a tiny p; with few, a large one may not reach 0.05. Whether an effect matters is read from its size, not its p-value.

A p-value above 0.05 means the data didn't rule out 'no effect'; it doesn't show there is none. A small study can miss a real difference. The range of effects it can't rule out says how much it actually told you.

When there is no effect, p < 0.05 still turns up 1 time in 20, by design. Run many tests and something will cross the line: with 20 tests of things that do nothing, the chance at least one looks significant is 64%.

Of the results with p < 0.05, the share that are real depends on how many of the ideas tested were real to begin with. When real effects are rare, a large share of significant results are flukes: with 10% real ideas and 80% power, 36% of them are.

Make it yours

Learn it with your own tutor.

Add it to your goals and Cognosc starts from what you already know, works through the rest with you, and keeps checking you still understand it.