Simpson’s Paradox: Better in Every Group, Worse Overall
How a treatment, hospital or university can look better in every group and worse overall. Simpson’s paradox, with kidney stones, Berkeley and baseball.
Hospital A saves more of its mild cases than hospital B. It also saves more of its severe cases. So overall, hospital A must save more patients.
It doesn’t have to. Hospital B can have the better overall survival rate, even though it does worse with every kind of patient. This is Simpson’s paradox, and when you first meet it, it feels like an arithmetic mistake. It isn’t. It’s one of the most important lessons in reading data, because it shows that an overall figure can point in exactly the wrong direction.
Try it
Better in every group, worse overall
Hospital A saves more patients than B in both groups. Change how many severe cases each one takes.
A is better for mild cases (95% against 90%) and for severe ones (60% against 50%), yet worse overall: 67% against 82%. It’s because A takes more of the severe cases. That’s Simpson’s paradox.
How it happens
The trick is in the mix. Suppose A is a big specialist hospital that takes most of the severe cases, and B takes mostly mild ones.
Severe cases have much lower survival rates at any hospital. So A’s overall rate is dragged down by all its severe cases, even though it does better than B with every kind of patient. B’s overall rate looks good because it mostly treats patients who would do well anywhere.
Nothing is wrong with the arithmetic. The overall figure is answering a different question: what happened to the patients each hospital happened to get? rather than which hospital does better for a given patient? Play with the toy at the top: change how many severe cases each hospital takes, and you can make the overall winner flip while the group-by-group results stay the same.
The kidney stones
The cleanest real case comes from a 1986 study of treatments for kidney stones. Doctors compared open surgery (call it A) with a less invasive procedure (call it B), and found this:
| Treatment A | Treatment B | |
|---|---|---|
| Small stones | 93% success (81 of 87) | 87% success (234 of 270) |
| Large stones | 73% success (192 of 263) | 69% success (55 of 80) |
| All stones | 78% success (273 of 350) | 83% success (289 of 350) |
A wins for small stones. A wins for large stones. B wins overall.
The reason is in the counts. A was used mostly on large stones, which are harder to treat by any method, and B mostly on small ones. Doctors gave the tougher cases to the more robust treatment. So A’s overall success rate is weighed down by the hard cases, and B’s is flattered by the easy ones.
Try it
The kidney stone reversal
Real success rates from a 1986 study. Treatment A beats B for small stones (93% against 87%) and for large stones (73% against 69%). What decides the overall winner is which patients each one got. Change the mix.
A is better for both kinds of stone, yet overall it scores 78% against 83%, because it was given far more of the hard cases. Even out the mix and the reversal disappears.
If you have a large stone, the question that matters to you is “which works better for large stones?”, and the answer is A. The overall figure would steer you the wrong way.
Try it
You have a large stone
Using that study, you have a large kidney stone. Treatment A succeeded with 73% of large stones and B with 69%. Overall, though, A succeeded 78% of the time and B 83%. Which should you prefer?
Pick one. You’ll see why straight after.
Berkeley, 1973
The most famous example involves people rather than patients. In 1973, the University of California, Berkeley looked at its graduate admissions and found that men had been admitted at a noticeably higher rate than women, about 44% against 35%. It looked like clear evidence of bias.
When statisticians Peter Bickel, Eugene Hammel and J. W. O’Connell examined the figures department by department, the picture changed. In most departments, women were admitted at similar or slightly higher rates than men. The overall gap came from where people applied. Women applied more often to departments that were very competitive and rejected most applicants of either sex. Men applied more often to departments with higher admission rates for everybody.
It’s worth being careful about what this does and doesn’t show. It shows the admissions decisions within departments weren’t biased against women. It doesn’t answer the deeper question of why women were steered towards fields with fewer places. Data can settle one question and leave the next one open.
Baseball, without the jargon
Sports fans meet the paradox in batting averages. In 1995, David Justice had a slightly higher batting average than Derek Jeter. In 1996, Justice was slightly higher again. But over the two years combined, Jeter was clearly ahead.
How? In 1995 Jeter barely played, so his low-ish average that year came from only 48 at-bats, while Justice’s came from over 400. In 1996 it was the other way round: Jeter played a full season and hit well, Justice played little. When you pool the years, each player’s average is dominated by the year he played most, and Jeter’s big year was a better year.
Build one yourself in a minute
The paradox gets less spooky once you’ve made one. Take two tutors, Ana and Ben, and a test that pupils pass or fail.
- Ana teaches 10 strong pupils and 90 who are struggling. 10 of the strong ones pass (100%), and 45 of the strugglers (50%). Overall: 55 of 100, or 55%.
- Ben teaches 90 strong pupils and 10 who are struggling. 81 of the strong ones pass (90%), and 4 of the strugglers (40%). Overall: 85 of 100, or 85%.
Ana has the higher pass rate with strong pupils, 100% against 90%, and with struggling ones, 50% against 40%. Ben has the far better overall rate, 85% against 55%. All it took was giving Ana most of the hard cases. Once you see that the overall rate is just a weighted mix of the group rates, and that the weights can be anything, the reversal stops being a mystery. It becomes a warning: never compare two overall rates without asking what they’re made of.
Spotting it in the wild
Simpson’s paradox can happen whenever two conditions hold:
- There’s a hidden grouping (severity, department, stone size, age) that strongly affects the outcome, and
- The things being compared have very different mixes of it.
When you see an overall comparison, ask: is there a factor that strongly affects the outcome, and do the groups differ a lot in it? If so, look at the figures broken down by that factor before drawing any conclusion.
It showed up early in the Covid-19 pandemic. In early 2020, the share of confirmed cases who died was higher in Italy than in China. But within every age group, the death rate was lower in Italy. Italy’s cases skewed much older, and age was by far the biggest driver of risk. The overall comparison was mostly a comparison of age profiles.
Which number should you believe?
This is where Simpson’s paradox becomes genuinely deep. It’s tempting to say the split figures are always the real answer and the pooled ones always mislead. That’s often right, but not always.
The right choice depends on why the groups differ. The computer scientist Judea Pearl, who has written a great deal about cause and effect, puts it this way: you have to know the causal story.
- Split on a confounder. If the grouping factor affects both which treatment someone gets and how they do, like stone size, compare within groups. Stone size isn’t caused by the treatment; it decided the treatment.
- Don’t split on something the treatment causes. Suppose a drug works partly by lowering blood pressure. If you split patients by their blood pressure after treatment, you’d be removing part of the drug’s effect from the comparison, and could conclude it does nothing when it works well. Here the pooled figure is the honest one.
The same numbers can call for different readings depending on how the world produced them. Statistics alone can’t tell you which; a sensible model of the situation can.
What to take from it
Whenever you see a single overall comparison between two hospitals, schools, treatments, cities or companies, ask what’s inside it. What kinds of cases did each one get? Would a fair comparison look at like against like?
A league table of surgeons that doesn’t adjust for how ill their patients were may simply rank them by how cautious they are about taking on hard cases. A school that admits mostly high-attaining pupils can top a table without teaching any better. And a treatment can look worse overall precisely because doctors trusted it with the hardest cases.
The overall number isn’t lying. It’s answering a question you didn’t ask. To practise the habit of asking the right one, try the free statistics test, and see correlation versus causation for the bigger family of traps this belongs to.
Questions people ask
- What is Simpson’s paradox in simple terms?
- It’s when something looks better in every group you split the data into, but worse when you add the groups together, or the other way round. It isn’t a mistake in the arithmetic. It happens because the groups have different mixes of something that matters a lot.
- What causes Simpson’s paradox?
- A third factor that strongly affects the outcome and is spread unevenly between the things being compared. A hospital that takes the hardest cases can be better at every kind of case and still have worse overall results, because the hard cases drag its average down.
- What is the Berkeley admissions example?
- In 1973 the University of California, Berkeley admitted a higher share of male than female graduate applicants overall. But within most departments women did as well or better. Women had applied more often to the most competitive departments, which turned most applicants away whatever their sex.
- Should you use the combined data or the split data?
- It depends on the cause. If the splitting factor affects both which group someone is in and the outcome, like stone size in the kidney stone study, compare within groups. If the factor is caused by the treatment itself, splitting on it can mislead, and the combined figure is the honest one.
- Is Simpson’s paradox common in real data?
- More common than people expect. It has appeared in medical trials, university admissions, sports statistics, tax data and Covid-19 death rates, which were higher overall in Italy than in China in early 2020 but lower in every age group, because Italy’s cases were older.
Sources and further reading
- Simpson, E. H. (1951). The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society, Series B, 13(2), 238–241.
- Bickel, P. J., Hammel, E. A. and O’Connell, J. W. (1975). Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175), 398–404.
- Charig, C. R., Webb, D. R., Payne, S. R. and Wickham, J. E. (1986). Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. BMJ, 292(6524), 879–882.
- Ross, K. (2004). A Mathematician at the Ballpark. Pi Press.
- von Kügelgen, J., Gresele, L. and Schölkopf, B. (2021). Simpson’s paradox in Covid-19 case fatality rates: A mediation analysis of age-related causal effects. IEEE Transactions on Artificial Intelligence, 2(1), 18–27.
- Pearl, J. (2014). Comment: Understanding Simpson’s paradox. The American Statistician, 68(1), 8–13.
