Andrew Gelman takes up the question of the “quals” versus the “quants.” To nobody’s surprise, Gelman has trouble with people throwing around numbers, and the tendency to use numbers as a claim to authority. Me, too. I recently read a sentence in a draft that somebody sent me to review:

In 2003, the Bureau of Transportation found that 6 million adults with disabilities felt that public transportation did not meet their needs. Further, 1.9 million adults with disabilities never leave their home because they lack access to transit (Feeley, 2009).

This type of sentence has to make you stop. I set out to do some basic fact-checking. There is no Bureau of Transportation, at least not in the US. And who is this Feeley person?

Turns out, Feeley is a consultant who wrote a very nice paper for the Transportation Research Board Annual Meetings in 2009 on autism and access [pdf link]–what I am writing about at the moment based on work that I had students collect. This Feeley paper should have gone into the literature review in the first place, but didn’t, serving only as secondary citation fodder for a point the authors wanted to make about the issue close to their heart: transit.

So I go find the original brief:

U.S. Dept of Transportation, Bureau of Transportation Statistics. Issue Brief – Transportation Issues Keep Over Half a Million Disabled at Home. April 2003. No. 3.

Suddenly, even in the title shows that the 1.9 million people “staying home due to lack of transit” wasn’t right. In my book, the estimated 560,000 people staying home is still awful–unless they like staying home. The brief shows that 1.9 million do stay at home, but they don’t do so just because of transportation. They do so for a variety of reasons.

The last bit is a tiny bit, but a big problem with lack of care. The original sentence claimed that lack of access to transit is what caused the (incorrect) 1.9 million to stay home. But the BTS brief actually describes the difficulties:

The one difficulty cited most frequently was the lack of a personal vehicle. Other difficulties cited by respondents included public transportation availability or cost, physical problems that made using transportation too difficult, and personal preferences, such as not wanting to ask others for help or having to depend on someone else for transportation.

That’s a much different description of the problem than “they lack access to transit.” In fact, the # 1 self-defined problem is not having a car.

There are host of disabilities for which transit is not necessarily a great big help (there are some that good-quality transit is a big help for, but not all, and high-quality transit…well, it’s pretty rare.)

So when does one stop being “in the ballpark” and start being just plain misleading in a way that subverts the truth?