Syllabus


With links to readings

Calling Bull in the Age of Big Data

Logistics

Course: INFO 198 / BIOL 106B. University of Washington
To be offered: Spring Quarter 2017
Credit: 1 credit, C/NC
Enrollment: 160 students
Instructors: Carl T. Bergstrom and Jevin West
Synopsis: Our world is saturated with bull. Learn to detect and defuse it.

The course will be offered as a 1-credit seminar this spring through the Information School at the University of Washington. We aim to expand it to a 3 or 4 credit course for 2017-2018. For those who cannot attend in person, we aim to videotape the lectures this spring and make video clips freely available on the web.


Learning Objectives

Our learning objectives are straightforward. After taking the course, you should be able to:

  • Remain vigilant for bull contaminating your information diet.
  • Recognize said bull whenever and wherever you encounter it.
  • Figure out for yourself precisely why a particular bit of bull is bull.
  • Provide a statistician or fellow scientist with a technical explanation of why a claim is bull.
  • Provide your crystals-and-homeopathy aunt or casually racist uncle with an accessible and persuasive explanation of why a claim is bull.

We will be astonished if these skills do not turn out to be among the most useful and most broadly applicable of those that you acquire during the course of your college education.


Schedule and readings

Each of the lectures will explore one specific facet of bull. For each week, a set of required readings are assigned. For some weeks, supplementary readings are also provided for those who wish to delve deeper.

Lectures

  1. Introduction to bull
  2. Spotting bull
  3. The natural ecology of bull
  4. Causality
  5. Statistical traps
  6. Visualization
  7. Big data
  8. Publication bias
  9. Predatory publishing and scientific misconduct
  10. The ethics of calling bull.
  11. Fake news
  12. Refuting bull

Week 1. Introduction to bull. What is bull? Concepts and categories of bull. The art, science, and moral imperative of calling bull. Brandolini’s Bullshit Asymmetry Principle.

Supplementary readings

  • G. A. Cohen (2002) Deeper into Bull. Buss and Overton, eds., Contours of Agency: Themes from the Philosophy of Harry Frankfurt Cambridge, Massachusetts: MIT Press.
  • Philip Eubanks and John D. Schaeffer (2008) A kind word for bull: The problem of academic writing. College Composition and Communication 59(3): 372-388
  • J. L. Austin Performative Utterance, in Austin, Urmson, and Warnock (1979) Philosophical Papers. Clarendon.

Week 2. Spotting bull. Truth, like liberty, requires eternal vigilance. How do you spot bull in the wild? Effect sizes, dimensions, Fermi estimation, and checks on plausibility. Claims and the interests of those who make them. Forensic data analysis: GRIM test, Newcomb-Benford law.


Week 3. The natural ecology of bull. Where do we find bull? Why news media provide bull. TED talks and the marketplace for upscale bull. Why social media provide ideal conditions for the growth and spread of bull.


Week 4. Causality One common source of bull data analysis arises when people ignore, deliberately or otherwise, the fact that correlation is not causation. The consequences can be hilarious, but this confusion can also be used to mislead. Confusing causality with necessity or sufficiency. Regression to the mean pitched as treatment effect. Milton Friedman's thermostat. Selection masked as transformation.

Supplementary reading

  • Karl Pearson (1897) On a Form of Spurious Correlation which may arise when Indices are used in the Measurement of Organs. Proceedings of the Royal Society of London 60: 489–498. For context see also Aldrich (1995).

Week 5. Statistical traps and trickery. Base-rate fallacy / prosecutor's fallacy. Simpson's paradox. Data censoring. Will Rogers effect, lead-time bias, and length time bias. Means versus medians. Importance of higher moments.


Week 6. Data visualization. Data graphics can be powerful tools for understanding information, but they can also be powerful tools for misleading audiences. We explore the many ways that data graphics can steer viewers toward misleading conclusions.

  • Edward Tufte (1983) The Visual Display of Quantitative Information Chartjunk: vibrations, grids, and ducks. (Chapter 5)
  • Tools and tricks: Misleading axes
  • Tools and tricks: Proportional Ink

Week 7. Big data. When does any old algorithm work given enough data, and when is it garbage in, garbage out? Use and abuse of machine learning. Misleading metrics. Goodhart's law.

Supplementary reading

  • Cathy O'Neil (2016) Weapons of Math Destruction Crown Press.
  • Peter Lawrence (2014) The mismeasurement of science. Current Biology 17:R583-585

Week 8. Publication bias. Even a community of competent scientists all acting in good faith can generate a misleading scholarly record when — as is the case in the current publishing environment — journals prefer to publish positive results over negative ones. In a provocative and hugely influential 2005 paper, epidemiologist John Ioannides went so far as to argue that this publication bias has created a situation in which most published scientific results are probably false. As a result, it’s not clear that one can safely rely on the results of some random study reported in the scientific literature, let alone on Buzzfeed.

Supplementary Reading


Week 9. Predatory publishing and scientific misconduct. Predatory publishing. Beall's list and his anti-Open Access agenda. Publishing economics. Pathologies of publish-or-perish culture. Pursuit of PR instead of progress.


Week 10. The ethics of calling bull. Where is the line between deserved criticism and targeted harassment? Is it, as one prominent scholar argued, “methodological terrorism” to call bull on a colleague's analysis? What if you use social media instead of a peer-reviewed journal to do so? How about calling bull on a whole field that you know almost nothing about? Pubpeer. Principles for the ethical calling of bull. The Dunning-Kruger effect. Differences between being a hard-minded skeptic and being a domineering jerk.


Week 11. Fake news.. Fifteen years ago, nascent social media platforms offered the promise of a more democratic press through decentralized broadcasting and a decoupling of publishing from advertising revenue. Instead, we get sectarian echo chambers and, lately, a serious assault on the very notion of fact. Not only did fake news play a substantive role in the November 2016 US elections, but recently a fake news story actually provoked nuclear threats issued by twitter.


Week 12. Refuting bull. Refuting bull requires different approaches for different audiences. What works for a quantitatively-skilled professional scientist won't always convince your casually racist uncle on facebook, and vice versa.