February 12, 2016

The failure of replication in economics (and health economics)

Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say “Usually Not”

One of the main principles of scientific method is reproducibility. Wikipedia says:
Reproducibility is the ability of an entire experiment or study to be duplicated, either by the same researcher or by someone else working independently. Reproducing an experiment is called replicating it.
The values obtained from distinct experimental trials are said to be commensurate if they are obtained according to the same reproducible experimental description and procedure. The basic idea can be seen in Aristotle's dictum that there is no scientific knowledge of the individual, where the word used for individual in Greek had the connotation of the idiosyncratic, or wholly isolated occurrence. Thus all knowledge, all science, necessarily involves the formation of general concepts and the invocation of their corresponding symbols in language (cf. Turner). Aristotle′s conception about the knowledge of the individual being considered unscientific is due to lack of the field of statistics in his time, so he could not appeal to statistical averaging by the individual.
A particular experimentally obtained value is said to be reproducible if there is a high degree of agreement between measurements or observations conducted on replicate specimens in different locations by different people—that is, if the experimental value is found to have a high precision
If this is so, then somebody could test any specific article and its results. That's precisely what two researchers have done with sixty published papers. The conclusion is specially annoying:
We successfully replicate the key qualitative result of 22 of 67 papers (33%) without contacting the authors. Excluding the 6 papers that use confidential data and the 2 papers that use software we do not possess, we replicate 29 of 59 papers (49%) with assistance from the authors. Because we are able to replicate less than half of the papers in our sample even with help from the authors, we assert that economics research is usually not replicable.
If economics research is usually not replicable, then what about health economics?

PS. Somebody should change immediately the peer review process or othewise close the journal.

PS.  Experience Ai Weiwei at the Royal Academy of Arts online with Ai Weiwei 360.