Ok, so my rant at AER prompted gentle readers to note some new items and some things we should consider. Cosma Shalizi, who writes the delightful blog, Three-Toed Sloth, notes that my ire towards AER is a mite misplaced, as the Reinhart-Rogoff manuscript appeared in one of the issues of proceedings done by AER of the economist’s national conference, and those papers are not peer-reviewed. Cosma pointed me to this excellent entry by Victoria Stodden, whose blog I clearly need to follow, on why Reinhart-Rogoff slipped through and how the state of the practice needs to change.

All of her suggestions are spot on, but I particularly like her idea around a site where reviewers can run code easily to replicate results:

This is typically nontrivial, since having the code and data doesn’t guarantee replication is either possible or achievable without significant effort. I have been working on a not-for-profit project called RunMyCode.org which could help reviewers by providing a certification that the code and data do regenerate the tables and figures in the paper. The site provides a web interface that permits users to regenerate the published results, and download the code and data

But still, HANDS ON HIPS, AER. YOU ARE CONSULTANTS TO POWER, A POSITION YOU HAVE CULTIVATED CAREFULLY. WHY does AER still get to hold the position it does as one of the “A” journals when it is publishing issues of papers that have not been reviewed…at all? Shouldn’t people be expected to put an asterix by the title of such an entry? I’m sorry–but DAYUM. If the sociologists did that, it would be yet further proof of their “junk research” in “junk journals.” And…the paper has been cited by economists in economics journals–nearly 500 times. Now, maybe all those cites are critical, but we are still dealing with the fact a paper which was NOT peer-reviewed IS CITED 500 TIMES. Help me understand how this demonstrates a discipline in which the utmost care is taken before policy prescriptions are advocated?

Remember, we are talking about a paper that has such big assumptions–unexplained weighting and dropped cases–that if one of my students presented such a paper in graduate seminar they would get public spanking.

Still more, a paper not peer-reviewed gets treated by the WashPo like it represents a consensus position among economists, no matter how often the paper cited or didn’t cite it. Gentle reader Jesse Richardson sent this op-ed to me, where the Op-Ed board of the WashPo demonstrates their innumeracy and makes excuses for lazy reporting. No, I don’t think Reinhart and Rogoff are responsible for global austerity, which is not global, btw. They saw an important research question in today’s most significant macro-economic debate, and they attempted to address it. That is, in general, the point of policy research, and I’m glad there are people attempting empirical verification of various approaches.

And moreover, academics don’t have that kind of influence. The reason why this paper got lots of play was the Havard + a finding that is convenient for a certain ideological position.

What I do think these researchers are responsible for is their analysis. If you are going to run with the big boys and claim to have a policy-relevant ratio or threshold for a big-deal policy question…you quadruple-check your G-D Excel and you go through a big demonstration of your robustness checks. It’s entirely possible that Reinhart-Rogoff’s original analytical choices are completely defensible. But dang it, in rigorous work, you use robustness checks to provide a range of where the threshold should be…and to preclude looking as though you have a scientifically proven law on your hands when what you really have is a model where your key choices inherently influence the outcomes.

I still stand by what I said yesterday. The researchers were lazy, the economic community was lazy, and the WashPo was lazy in citing this manuscript as though the percentages and ratios cited therein represented some meta-analytical finding resulting from careful aggregation of decades of work. It’s intellectually sloppy, and it’s counterproductive to real policy inquiry and democratic dialogue.

RRRRrrrrrrrrRRRRRR. Like empirical research into macro and macro policy AREN’T HARD ENOUGH, PEOPLE.