Case Studies


Bull in the wild

Spotting bull in the wild it isn't something you have to let others do for you. To illustrate this, we've provided a set of case studies based upon examples of bull in the wild. We've spotted many of these ourselves; some have come via other channels. These cases aren't the most egregious examples out there, but each illustrates one or more of the principles and practices that we aim to teach in this course. We will be adding additional case studies on a regular basis.

Case studies

Basic

These case studies require only clear thinking and occasionally a bit of arithmetic. As such, they should be readily accessible to all of our readers.

  • Food stamp fraud. In this example drawn from a Fox News story, we demonstrate how Fermi estimation can cut through bull like a hot knife through butter.

  • Traffic improvements. Irrelevant facts can lead you to bull conclusions if you approach them with an inaccurate model of how the world works. And if those conclusions let you spin a trite story about terrible traffic and wasteful government expenditures, what better clickbait?

  • 99.9% caffeine-free. In one section of his book The Demon-Haunted World, Carl Sagan decried that way that advertisers try to dazzle us with irrelevant facts and figures. He was mostly concerned with drug advertising; in this case study we explore a more innocuous example.

  • Criminal machine learning. Machine learning algorithms are sometimes touted as generating results that unbiased and without prejudice. This is bull. We explore an example in which two authors claim to have an algorithm that can determine whether one is a criminal simply from an 80x80 facial image, and show that this algorithm is actually doing something very different.

Intermediate

These case studies introduce some basic concepts from statistics such as sample size and extrapolation. However, they do not require any technical statistical knowledge to follow and most readers should find these relatively accessible.

  • Track and field records as examples of senescence. We lead off our series of case studies by calling bull on a figure in one of our own publications. We explore how differences in sample sizes can create misleading patterns in data, and an example of how writing a simulation can be an effective method of calling bull.

  • A gender gap in 100 meter dash times. We examine a 2004 Nature paper predicting that women sprinters will outrun men by the mid-22nd century. In doing so, we see the danger of over-extrapolation, and we get to read a beautiful example of reductio ad absurdum as a means of calling bull.

  • Musicians and mortality. Here we consider what can go wrong as one goes from scholarly article to popular science piece to social media meme. We explore why a data graphic shared widely on social media gives a misleading impression, explain the issue of right-censoring, and discuss how its effects can be seen in the light of correlation analysis.

Advanced

These case studies make extensive use of calculus and/or mathematical statistics. They may be of interest to readers without a strong background in those areas, but they will be most accessible to readers who know a bit of calculus and/or statistics.

  • NIH's Rule of 21 - Part 1. The NIH wanted to restrict the number of grants that a single investigator could hold, and tried to justify this policy using data visualizations purported to illustrate decreasing marginal returns of investment in a given lab. Here we show that their graphs fail to demonstrate this and explain where they went wrong.