Understand Survivorship Bias To Make Better Decisions

While it’s natural to try to learn from success, it can provide a skewed outlook. We need to understand survivorship bias to make better decisions.

Survivorship bias happens when we base our understanding only on the experiences of those who succeed or are highlighted in some way, and do not take into account others who failed or are not visible. Formally, survivorship bias is a type of selection bias in which the results of a particular outcome (i.e., the survivors) are disproportionately evaluated, leading to an incorrect estimate of probability. In The Black Swan, Nassim Taleb uses the term silent evidence to describe the data obscured by survivorship bias.

For example, the popular media glorifies movie stars, athletes, and CEOs who dropped out of school but became successful. As a result, some people could conclude that formal education is unnecessary. However, the media rarely mention that a much larger number of dropouts with similar skills and determination don’t become as successful – often due to factors beyond their control.

The term survivorship bias was coined by Abraham Wald, who helped the American military reduce the likelihood airplanes would be shot down. The military wanted to protect planes with armor, but adding armor to the entire plane would make it too heavy to fly. It seemed natural to protect the portion of the plane which got shot the most often. A simple analysis showed most of the bullet holes in the planes which returned from combat were around the body and wings.

However, Wald showed that adding armor to the body and wings would have been a mistake. First, these are the largest parts on a plane which means statistically they should have the most bullet holes. More importantly, the initial analysis was based on planes that made it back safely and therefore subject to the survivorship bias. When they examined planes shot down in combat, they recognized the most vulnerable part was the engine. While engines might not have been hit frequently, when they were hit, the results were usually catastrophic. The armor was added to engines, reducing the number shot down in combat.

The key to avoiding the survivorship bias when making a decision is to ask yourself what’s missing in the information you have. It’s more than just being open minded but actively searching out additional – potentially contrary – data. In sales, win reports are commonly used but they ignore the reality that most companies lose more deals than they win. For most companies, there is more useful information in loss reports than in win reports (an example of Taleb’s silent evidence).

Not surprisingly, most marketing claims are subject to the survivorship bias. Weight-loss commercials tout clients who lost 10, 20, even 30 pounds using their methods but fail to mention that most people lose little if any weight. The Truth In Advertising Act forces advertisers to include a “results are not typical” disclaimer but most viewers miss this silent evidence because it’s in a tiny font or only displayed momentarily.

It’s tempting to only learn from success, but failure can be an important teacher as well. If you understand survivorship bias, you can make better decisions.