Thinking · 3 min read
Survivorship Bias: The Planes That Didn't Come Back
Why studying only the winners leads you wrong: the famous story of the WWII bombers, plus survivorship bias in business advice, investing and old buildings.
By the Cognosc team ·
During the Second World War, analysts looked at the bombers returning from missions and mapped where they’d been hit. The holes clustered on the wings, the tail and the body. The obvious conclusion: add armour where the holes are.
The statistician Abraham Wald, working with the Statistical Research Group in New York, pointed out the flaw. These were the planes that came back. A plane hit in the wings could still fly home. The places with no holes, like the engines, were where hits brought planes down, and those planes weren’t there to be counted. The armour belonged where the holes weren’t.
That’s survivorship bias: drawing conclusions from the things that made it through, while the things that didn’t are invisible.
Try it
Where should the armour go?
Send out bombers. Every part of a plane is equally likely to be hit. Then look at the holes on the planes that return.
Send out a squadron.
Why it’s so easy to fall for
The data you can see is real. The holes really are on the wings. The problem is what the data can’t show you, and absence is hard to notice. Nobody interviews the startups that failed, reads the novels that never got published, or tours the buildings that fell down.
Where it shows up
Success advice
Books about successful companies and people often list what they did: took big risks, dropped out of college, ignored the doubters. But many people who took big risks and ignored the doubters failed, and they don’t write books. Without the failures, you can’t tell whether the strategy caused the success or was just common among everyone who tried.
Investing
The average performance of investment funds looks better than it really was, because funds that performed badly are often closed or merged, and disappear from the records. Looking only at funds that still exist overstates how well funds do.
“They don’t make them like they used to”
Old buildings, furniture and cars often seem better made than new ones. Partly, that’s because the badly made old ones fell apart long ago. We only see the best of the past next to the whole range of the present.
Hardy survivors
The idea that what doesn’t kill you makes you stronger can come from the same effect: if a hardship kills the most vulnerable, the survivors look tough, even if the hardship made nobody stronger.
How to guard against it
- Ask what’s missing. For any group you’re studying, ask what process decided who ended up in it.
- Look for the failures. Before copying what winners did, check how many people did the same and lost.
- Be wary of stories told only by the survivors.
Test yourself
The bomber question is in the free statistics test, and the thinking traps test covers more ways missing information leads us astray.