4 out of 5 People Can Be Wrong

The law of small numbers (or hasty generalization) is the tendency to jump to a conclusion without enough evidence. In statistics, it’s called bias from insensitivity to sample size – generalizing from a limited number of events (a sample) selected from a much larger number of events (the population). For example, if a mutual fund manager has three above-average years in a row, you might conclude that the fund manager is better than average and will have a fourth above-average year. While it may be true, you cannot come to this conclusion from such a small amount of data.

This bias shows up frequently in the media. Imagine the headline summarizing a telephone survey of 500 rural inhabitants, in which 70% responded they didn’t have enough money for retirement. It would likely be something like this: People living in the country can’t comfortably retire. Most readers would ignore the details of the survey, including whether 500 people is sufficient to warrant the conclusion.

This is the challenge with small sample sizes. If the survey had only included 50 people, you’d be suspicious. And if it included 50,000 people, you probably wouldn’t be worried. But what about 500? Our intuition is good at the extremes but not in the middle. Without more information, it’s hard to know if the 500-person sample size is sufficient.

In the book Thinking, Fast and Slow, Daniel Kahneman elaborates:

The exaggerated faith in small samples is only one example of a more general illusion – we pay more attention to the content of messages than to information about their reliability, and as a result end up with a view of the world around us that is simpler and more coherent than the data justify. Jumping to conclusions is a safer sport in the world of our imagination than it is in reality.

Here’s a test of your sensitivity to sample size, courtesy of Max Bazerman in Judgment in Managerial Decision Making: A town has two hospitals; 45 babies are born each day in the larger one and 15 are born in the smaller one. Overall, about 51% of babies are boys but of course the exact percentage varies day to day. For one year, both hospitals tracked the number of days in which more than 60% of the babies born were boys. Which hospital had more of these days?

A. The larger hospital
B. The smaller hospital
C. About the same (within 5% of each other)

Did you guess C? Most people incorrectly choose C when the right answer is B. Having 60% boys in one day is a rare event and statistics tell us that we’re more likely to observe a rare event in a small sample than in a large one.

You can also see this effect in sports. In a sport with a long season like basketball, hockey or baseball, the standings 20 games into the season may not be representative of the final standings. But at the end of the season, the best team usually has the best record. This effect is why sports fans are often optimistic; on any given day, anything can happen – especially in the playoffs. As Michael Lewis writes in Moneyball: “In a five-game series, the worst team in baseball will beat the best about 15% of the time.”

Which brings us to advertising. The classic commercial which claims “4 out of 5 dentists recommend…” has been mimicked many times over the years. The law of small numbers teaches us this claim is meaningless unless we know the sample size.

My guess is that there were only 5 dentists. Which means they could be wrong.