Simpson's Paradox: Better in Every Group, Worse Overall

Hospital A saves more of its mild cases than hospital B. It also saves more of its severe cases. So overall, hospital A must save more patients, right?

Not necessarily. Overall, B can come out ahead. This is Simpson’s paradox, and it has fooled courts, universities and medical researchers.

How it happens

The trick is in the mix. Suppose A is a big specialist hospital that takes most of the severe cases, and B takes mostly mild ones.

Severe cases have much lower survival rates at any hospital. So A’s overall rate is dragged down by all its severe cases, even though it does better than B with every kind of patient. B’s overall rate looks good because it mostly treats patients who were likely to survive anyway.

Nothing is wrong with the arithmetic. The overall figure is just answering a different question: “what happened to the patients each hospital happened to get?” rather than “which hospital does better for a given patient?”

A famous example

In 1973, the University of California, Berkeley, looked at its graduate admissions. Overall, men were admitted at a noticeably higher rate than women, which looked like bias against women.

But department by department, women were admitted at similar or slightly higher rates. The difference came from which departments people applied to: women applied more often to highly competitive departments that rejected most applicants of either sex, while men applied more to departments with higher admission rates. Combined, the numbers told the opposite story to the one in each department.

Another: kidney stones

A well-known 1986 study compared two treatments for kidney stones. One treatment had a better success rate for small stones and for large stones, but a worse success rate overall, because it had been used mostly on the harder, large stones.

How to spot it

Simpson’s paradox happens when:

1. There’s a hidden grouping (severity, department, stone size) that affects the outcome strongly, and
2. The groups being compared have very different mixes of it.

So whenever you see a comparison of overall rates, ask:

• What groups are being pooled together?
• Do the things being compared have the same mix of those groups?
• Does the comparison hold within each group?

Which number is right?

It depends on the question, and usually on what causes what. If severity affects both which hospital you go to and whether you survive, you should compare within severity groups: that tells you which hospital is better for a patient like you. The pooled number mixes the hospital’s quality up with the kind of patients it gets.

This is the same problem as a confounder in correlation vs causation: a third factor tangled up with both the thing you’re comparing and the outcome.

Test yourself

Simpson’s paradox is one of ten ideas in the free statistics test.