Sometimes bull comes not in the form of misleading data, but rather in the form of misleading models — explicit or implicit.
In October 2016, Seattle's KOMO News ran a scathing story about a major road improvement project intended to relieve traffic congestion in the city. "74 million later, Mercer Mess is 2 seconds faster" screamed the headline. Here's the introduction to the story:
The implication is obvious: yet another colossal waste of taxpayer money.
And indeed these numbers sound pretty bad, especially if you ignore the fact traffic along this corridor exploded over the duration of the project due to massive growth of Amazon and numerous other tech companies in the South Lake Union area. If we approach the situation with a mental model in which each Seattle commuter is assigned a specific route which her or she follows faithfully, the $74 million in improvements don't seem to have served the commuters along the "Mercer Mess" route very well at all.
But pause a minute and think about how you expect traffic to flow in a big city during rush hour. The key is to think about traffic patterns as resulting from individual route choices made by tens of thousands of drivers. All of downtown Seattle experiences massive congestion during rush hour. The tight urban grid provides every driver with dozens of possible routes. Drivers that use these roads during rush hour are typically commuters who drive five days a week, month after month. They learn to adjust both their route and their travel times (where possible) in response to traffic patterns. As a result, we expect traffic patterns within the city to be some reasonable approximation of an equilibrium of the multiplayer game of "pick a route to get to work as fast as possible, given the routes that everyone else has picked."
In this equilibrium model, improvements to travel times on a given route will transiently make that route faster and thus more attractive than neighboring routes. Additional commuters will shift to the newly improved route until travel times are again equalized. Under this model, that vast majority of the improvement in travel times on the enhanced road should be absorbed by increased traffic there. Indeed, this is exactly what happened in Seattle, as the KOMO article later admits. The Mercer corridor now handles 30,000 more cars a day than it did before the improvement project, without any increase in travel time. The headline might have read "Seattle road improvement project allows 10 million extra trips per year with no increase in travel times".
Once we shift our thinking from a naive model in which improvements to a road affect only drivers on that road, to more reasonable model in which traffic patterns form a dynamic network across a city, it becomes clear that it's silly to quantify the benefits of improvements to one road by looking at travel times on that road. We can see that to measure quantify benefits, we need to integrate the consequences of an improvement project across all travels routes in that region of the city.
A skeptic might ask why, for $74 million, you couldn't improve travel times everywhere by a substantial amount. In a modestly-sized town you might indeed be able to do it. But in a big city like Seattle with serious traffic issues and massive urban growth...well, remember that a decade back Boston spent 15 billion dollars on its "Big Dig." While Boston traffic is better than it used to be, the problems are far from eliminated. Unfortunately, $70 million dollars is small potatoes when it comes to fixing the traffic woes of a rapidly growing metropolis. If the use of those funds can keep travel times constant in South Lake Union for a few years, it's money well spent in our view — and we don't even have to drive there on a daily basis.
Silly as this story might have been, we've got to hand it to KOMO News. Rather than firing back or (more likely) ignoring us entirely, they gave our project a very positive presentation in a recent news story. We're sorry, KOMO, that the FCC makes you blur out a photo of our web page. That's just....well, bull.
Some readers have questioned our bull call here. The argument seems to go something like this: "KOMO's headline that travel times have only decreased by 2 seconds is just as true as your proposed headline that 10 million extra trips occur yearly without increasing travel times. How can you call bull on something that is true?"
Good question. How can you call bull on something that is true? The answer is that true statements can be bull if they distract you from more important facts or give you the wrong impression about an overall situation. In our view, the KOMO headline does both. KOMO is purportedly a news outlet, not a compendium of random facts. But the minimal decline in travel time is hardly news; what else would you expect to happen if you were to think for a few seconds about how traffic works in this city? KOMO has omitted a true fact that is actually newsworthy (10 million extra trips a year without increasing travel time) in favor of one that is not (2 second decline in travel time).
We cannot resist an analogy. In early 2010, the Seattle Mariners renewed the contract of pitching ace Felix Hernandez for about $78 million dollars — comparable to the cost of the construction work on the Mercer Mess. But the Mariner's team batting average dropped from .258 in 2009 to .236 in 2010, and team home runs dropped from 160 to 101. KOMO could have run a headline "Mariners hitting declines in 2010 despite $78 million investment in Felix Hernandez". It's a true claim. But it's obviously a silly headline, because it couples two unrelated facts. The hitting decline has nothing to do with Hernandez. Pitchers play only every fifth game or so, they don't bat in the American league, and most importantly they are not signed for their batting prowess! Pairing Felix's contract with the team's 2010 hitting performance implies that Felix's salary was wasted because he didn't raise the team's batting average. Stupid, but not so different from suggesting that the Mercer Mess construction was a waste because travel times did not appreciably decrease. Both use irrelevant measures to evaluate the efficacy of a 70 million dollar investments in Seattle institutions.