The Baloney Detection Kit

Over the holiday break, I read ‘The Demon-Haunted Book: Science as a Candle in the Dark’ by Carl Sagan. The book is not nearly as well-known as ‘Cosmos‘ but should be. It introduces the scientific method to non-scientists and provides advice on how to apply critical thinking. In Sagan’s words:

What skeptical thinking boils down to is the means to construct, and to understand, a reasoned argument and – especially important – to recognize a fallacious or fraudulent argument.

The above quote is from my favorite section called ‘The Fine Art of Baloney Detection’ in which Sagan introduces a series of tools to identify bad arguments and construct better alternatives. Among other things, we are reminded not to confuse correlation with cause, to avoid weasel words, and to watch for slippery slope arguments. Sagan also cautions against falling prey to the confirmation bias; a common issue among executives:

Don’t get too attached to your own ideas, lest you get reluctant to reject them even in the face of evidence to the contrary.

Inspired by Carl Sagan’s ideas, Michael Shermer, the editor of Skeptic Magazine, proposed a 10-point checklist to assess the believability of a claim. Watch a video of him explaining:

For those who don’t have the time or inclination to watch the entire video, here’s an abbreviated version of the Baloney Detection Kit:

  1. How reliable is the source of the claim?
  2. Does the source make similar claims?
  3. Have the claims been verified by somebody else?
  4. Does this fit with the way the world works?
  5. Has anyone tried to disprove the claim?
  6. Where does the preponderance of evidence point?
  7. Is the claimant playing by the rules of science?
  8. Is the claimant providing positive evidence?
  9. Does the new theory account for as many phenomena as the old theory?
  10. Are personal beliefs driving the claim?

The last one, in particular, is a good reminder for all of us. Everyone, including scientists, have inherent biases which might color interpretation of data and impact decisions. To avoid these biases showing up in academic papers, scientists follow a peer review process which requires an (often anonymous) community of experts to provide an impartial review.

Maybe those of us in business should try something analogous. The next time you’re about to start a new project, ask someone with a completely different perspective to do a detailed peer review.

Just remember to keep an open mind to a contrary point of view.

, , , ,

7 Responses to The Baloney Detection Kit

  1. Peter January 11, 2016 at 6:33 am #

    Great post again! The last full paragraph is one of the best bits of advice you can take on anything important. Great post, thank you.

    • Jonathan Becher January 13, 2016 at 10:39 am #

      Thanks Peter. Many people pride themselves on open door policies. We should also pride ourselves on open mind policies.

  2. oliviafalkenstein January 19, 2016 at 2:44 am #

    Found this very useful, thank you Jonathan. One of the next reads for me will also be a Carl Sagan book. As a vegan / idealist I am often confronted with radical, new age-like theories about health, healing, vaccination etc so checking the sources and having a critical-skeptical attitude has proven very important.
    One question is: how does one deal with kids? should we indulge in nice stories about Santa Claus, Easter Bunny, Toothfairy and so on? 🙂 or do we start “telling the truth” early on?


  1. The Confirmation Bias Is No Joke - Manage By Walking Around - December 9, 2018

    […] can combat fake news through rigorous fact-checking and use of the baloney detection kit but overcoming a confirmation bias is even more difficult. When I have a strong point of view, I […]

  2. The Many Sides of the Truth - Manage By Walking Around - October 6, 2019

    […] better than others. For me, I combat fake news through rigorous fact-checking and use of the baloney detection kit. When I have a strong point of view, I try to be open-minded by asking myself what would […]

  3. Don’t Confuse Me With Facts - Manage By Walking Around - December 13, 2020

    […] there are many reliable techniques to make sure you have an open mind, it’s much more difficult to apply these techniques to others. […]

  4. The Green Coffee Bean Problem - Manage By Walking Around - April 7, 2024

    […] deep critical thinking is time-consuming and difficult. But often we don’t even do a cursory inspection of claims. […]

Leave a Reply