Case Studies

Bull in the wild

Spotting bull in the wild it isn't something you have to let others do for you. To illustrate this, we've provided a set of case studies based upon examples of bull in the wild. We've spotted many of these ourselves; some have come via other channels. These cases aren't the most egregious examples out there, but each illustrates one or more of the principles and practices that we aim to teach in this course. We will be adding additional case studies on a regular basis.

Case studies

  • Track and field records as examples of senescence. We lead off our series of case studies by calling bull on a figure in one of our own publications. We explore how differences in sample sizes can create misleading patterns in data, and an example of how writing a simulation can be an effective method of calling bull.

  • Food stamp fraud. In this example drawn from a Fox News story, we demonstrate how Fermi estimation can cut through bull like a hot knife through butter.

  • Musicians and mortality. Here we consider what can go wrong as one goes from scholarly article to popular science piece to social media meme. We explore why a data graphic shared widely on social media gives a misleading impression, explain the issue of right-censoring, and discuss how its effects can be seen in the light of correlation analysis.

  • Traffic improvements. Irrelevant facts can lead you to bull conclusions if you approach them with an inaccurate model of how the world works. And if those conclusions let you spin a trite story about terrible traffic and wasteful government expenditures, what better clickbait?

  • 99.9% caffeine-free. In one section of his book The Demon-Haunted World, Carl Sagan decried that way that advertisers try to dazzle us with irrelevant facts and figures. He was mostly concerned with drug advertising; in this case study we explore a more innocuous example.

  • A gender gap in 100 meter dash times. We examine a 2004 Nature paper predicting that women sprinters will outrun men by the mid-22nd century. In doing so, we see the danger of over-extrapolation, and we get to read a beautiful example of reductio ad absurdum as a means of calling bull.

  • Criminal machine learning. Machine learning algorithms are sometimes touted as generating results that unbiased and without prejudice. This is bull. We explore an example in which two authors claim to have an algorithm that can determine whether one is a criminal simply from an 80x80 facial image, and show that this algorithm is actually doing something very different.