Survivorship Bias: The Planes That Didn’t Come Back

In 1943, the American military had a problem it could measure. Bombers were flying out over Europe and too many weren’t coming back. Armour plating would help, but armour is heavy, and a heavier plane flies shorter and slower and carries fewer bombs. So the question was where to put the weight.

The Air Force had data. Engineers inspected the planes that made it home and mapped every bullet hole. The pattern was clear: holes clustered across the wings, the tail and the main body of the fuselage. The engines and the cockpit were hit far less. The obvious plan was to bolt armour to the parts that were being shot up.

Then someone took the data to a statistician named Abraham Wald, who worked in a classified group at Columbia University, and Wald said something that sounds almost too simple. Put the armour where the holes aren’t.

His reasoning was this. These were the planes that came back. A plane with a hole in its wing could still fly home. A plane with a hole in its engine often couldn’t. The engines looked safe only because the planes that were hit there were lying in fields in Germany, absent from the sample. The clean engines weren’t good luck. They were a body count.

How the same hits give different holes

Suppose you could watch every hit, not only the holes on the survivors. You’d see bullets landing roughly everywhere, and the engines taking as many as the wings. The pattern flips only when you add the filter: you get to see a plane only if it survived.

That’s the whole mechanism. Survivorship bias is what you get when the thing you’re studying has been run through a filter that removes the failures, and you forget the filter is there. The data isn’t wrong. Every hole on those returning planes was really there. The error is in the question you think the data answers.

The trick for spotting it is one question, and you can ask it of almost anything: what had to be true for this to be in front of me? A plane had to survive. A company had to stay in business. A book had to find a publisher. A building had to stay standing. Whatever the filter was, the things you can see are the ones that got through it.

Where it’s hiding in everyday life

Success advice

Take any shelf of business books. They study winners, find what the winners had in common, and offer it as a recipe: they took big risks, they ignored the doubters, they dropped out, they got up at five.

Now ask the Wald question. How many people took big risks, ignored the doubters, and lost everything? How many people got up at five and still went bust? They’re not in the book, because nobody writes a case study about a business that quietly closed. Yet they’re far more numerous than the winners. If a thousand people take the same long shot and one succeeds, that one will give talks about courage. The strategy might be sound, or it might be a lottery ticket. You can’t tell from the survivor alone.

None of this means winners learned nothing. It means a pattern found only among winners tells you what they did, not what made the difference. To learn from them, you need the losers who did the same things, so you can compare.

Investing

Fund managers love to report the average performance of their funds. The catch is that poor funds tend to be closed or merged into better ones, and when they go, they usually vanish from the databases too. What’s left is a list of survivors.

In a well-known study of US mutual funds, Elton, Gruber and Blake estimated that survivorship bias overstated returns by roughly a percentage point a year. That sounds small until you compound it over decades. It’s the difference between a retirement pot that looks comfortable on paper and one that’s smaller in reality.

The same filter hides in “this fund beat the market ten years running”. Start a thousand funds, let luck do its work, and some will have a ten-year streak. Close the rest and you can print their record in an advert.

“They don’t make them like they used to”

Old buildings look sturdier than new ones, old furniture looks better made, old cars look more charming and more solid. There’s something to that, because good craftsmanship does last. But look at what you’re comparing: a few centuries’ worth of the best-built survivors against everything built last year, good and bad alike. The jerry-built terraces and rickety chairs of 1850 fell apart long ago and have been quietly removed from the sample. The past looks like it was built better partly because only the well-built parts of it are left.

“What doesn’t kill you makes you stronger”

A harsh upbringing, a brutal training regime, a crash diet. Some people who went through it are tougher and say it made them so. But if a hardship removes the people who couldn’t take it, then everyone you meet afterwards looks as though they thrived. The same logic hides in “my grandfather smoked and lived to 95”. Of course you’ve heard from the one who lived to 95. The ones who died at 60 aren’t around to tell you the story.

Science and the news

Scientific journals publish studies that found something. A study that tried a promising treatment and found it did nothing is far less likely to be written up, let alone printed, so it ends up in a drawer. The published literature is a survivor list. If twenty teams test the same idea and one gets an exciting result by luck, that is the one you’ll read about. This is called the file-drawer problem, and it is one reason a surprising finding should be treated as a lead, not a fact, until someone else gets the same result.

The news does it too. Planes landing safely aren’t a story; one that crashes is. That cuts the other way, and gives a different bias, because rare dramatic events get so much attention that they feel more common than they are. Both come from the same place: what reaches you has been chosen, and not by chance.

Reviews, rankings and “top 10” lists

Restaurants that closed aren’t on the review site. Apps that failed aren’t in the store. Hedge funds that blew up aren’t in the ranking. Any list of what currently exists has been filtered by what survived long enough to be listed, and that makes the average of the list look better than the average of what was ever tried.

When it’s the opposite of what you’d expect

The most interesting thing about Wald’s insight is that it ran against the evidence everyone could see. When data tells you where the damage is, the instinct is to protect the damaged parts. His point was that, in this sample, absence was the signal. The places with no holes were the places where a hit was fatal.

That flips a common intuition: that more evidence means more certainty. If the evidence is filtered, more of it can make you more wrong, because you become surer of a picture that’s missing a piece. Engineers use this idea whenever they study crashes, not just flights that landed safely. It’s the reason aviation investigates every accident, and why a field with a culture of examining its failures tends to improve faster than one that celebrates its successes.

How to catch it before it catches you

You can’t fix a filter you haven’t noticed, so most of the work is noticing. A few habits help:

• Ask who’s missing. For any group you’re studying, ask what process decided who ended up in it. Who dropped out, closed, died, was deleted or never got published?
• Find the denominator. “Seventeen of twenty billionaires” means little. “Seventeen of every thousand people who tried” would mean a lot. Whenever you see a count of winners, look for the count of everyone who tried.
• Go looking for the graveyard. Failed startups, closed funds, studies that found nothing and never got published. They exist, even if they’re harder to find.
• Distrust “always” and “all of”. All successful people do X is almost always a sentence about the survivors.
• Compare with the people who did the same thing and failed. That’s the only way to see whether the strategy helped or whether it was just common.

A harder case: the school that “produces” success

Imagine a school whose alumni include a prime minister, three Olympians and a Nobel winner. Does the school make people successful? Maybe. But the school also chooses who it admits. If it takes only children who were already bright, ambitious and well-connected, those children would likely have done well anywhere, and the school’s record is partly just a record of whom it picked. Add that you hear about the famous alumni and not the ones who drifted into jobs nobody writes about, and the school’s halo grows brighter still.

To work out what the school adds, you’d need to compare its pupils with similar pupils who went elsewhere. That’s harder than counting famous names, which is exactly why people count famous names.

What it doesn’t mean

It doesn’t mean success stories are worthless or that you can’t learn from people who made it. It means you should learn from them carefully: take their stories as hypotheses worth testing, not as proof.

It also doesn’t mean everything is a lottery. Some things really do separate winners from losers. Survivorship bias just stops you from knowing which, until you look at both sides. When someone offers you a pattern, the useful response isn’t cynicism, it’s a question: compared with what, and who’s missing?

And there’s a quiet upside. Once you’ve seen it, you notice it everywhere: in the ranking that only shows today’s leaders, in the “classic albums” of a decade that only kept the good ones, in the older relative whose habits “never did them any harm”. It makes the world slightly less impressive and a lot more intelligible, which is a fair trade.