The Problematic "famous file-sharing paper"

Craig Newmark has previously covered the problems with Felix Oberholzer-Gee and Koleman Strumpf's widely cited and discussed paper in the Journal of Political Economy entitled "The Effect of File Sharing on Record Sales: An Empirical Analysis." Stan Liebowitz had found numerous errors in the paper, but was not given access to their data. Unfortunately, under Steven Levitt's editorship at the JPE, this general type of claim has arisen previously. Anyway, Craig has a follow up posting on all this. Florenz Plassmann has translated an article from Handelsblatt, a leading German financial publication. This is what Craig posts from the article:

Recently, this paper has sparked a heated discussion. The relevance of the debate extends far beyond the paper in question. It questions the reliability of empirical studies in economics, and may ultimately challenge the way in which the crème de la crème of scientific journals deals with scientific evidence.

The key question is: how can a study that is based on secret data that nobody has double-checked be printed without close examination by one of the most prestigious economics journals? This is especially puzzling because the supplier of the data has a special interest in a certain result. The study of the two economists from Harvard and Kansas is based on proprietary data on music downloads, which the authors received from the file sharing services "MixmasterFlame" and "FlameNap."

. . . .

Liebowitz knew of the filesharing study before it was published because it had been circulated as a working paper. In his letter he told Levitt that, despite repeated requests, the authors did not provide him with an opportunity to check their results. Could he please use his influence as editor of the "JPE" to make such checks possible? Levitt declined to tell Handelsblatt whether he followed up on this request.

It appears that he did not. Even one year after publication, the authors still keep their data to themselves. Oberholzer-Gee told Handelsblatt that they had to sign an agreement not to share the data to get them from the file sharing service. The authors argued that they had to "protect their sources" and declined to provide Handelsblatt with either a copy of the agreement or the name of a reference at the file sharing service who could confirm their version.

Liebowitz pressed Levitt, the editor of the "JPE," to at least correct several mistakes and ambiguities before publishing the paper.

For example, the authors write that about half the reductions in music CD sales are the result of the increase in market share of music discount stores with smaller inventories. Liebowitz argues that this cannot possibly be correct. He calculates that, even under extreme assumptions, the reduction in inventories can at most account for one-sixth of the decrease in sales. "It is unbelievable that a top-journal like the "JPE" would publish such claims without any evidence," Liebowitz complains in his letter, and he points Levitt to an entire series of additional errors or ambiguities.

Levitt forwarded Liebowitz’ letter to the authors, who ignored it—their study was published with only minor changes. Since then, file sharing services can refer to an academic paper in one of the top economics journals to defend themselves against the music industry.

In principle, like many other journals, the "JPE" requires that authors publish not only their results but also disclose the data and the methods that they use to derive them. However, this requirement does not apply to Oberholzer-Gee and Strumpf—their paper was accepted before the requirement became binding. "This has nothing to do with science," criticizes Bruce McCullough, professor of decision sciences at Drexel University in Philadelphia. "Without scrutiny, there can be no science," says the expert on the replicability of empirical results in economics. . . .

Given the huge coverage of this by the American press, it would be useful if someone in the American Press would write about this problem.

Thanks very much to Florenz Plassmann for translating this.