Peer-reviewed publications are currently the main currency for academic advancement. Like fiat currencies in general economic use, the value of editorial peer review appears to be largely in its value for economic transactions. Consider, for example, a scholarly study published in 1982. The authors created fictitious authors for twelve articles published in twelve highly regarded psychology journals eighteen to thirty-two months earlier. These articles were then resubmitted for publication to the same journals that had published them previously. The outcome:
Of the sample of thirty-eight editors and reviewers, only three (8 percent) detected the resubmissions. This result allowed nine of the twelve articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the eighteen referees (89 percent) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.”[1]
A critical article on peer review recently noted:
At the British Medical Journal we took a 600 word study that we were about to publish and inserted eight errors. We then sent the paper to about 300 reviewers. The median number of errors spotted was two, and 20% of the reviewers did not spot any. We did further studies of deliberately inserting errors, some very major, and came up with similar results.[2]
Methodological and statistical analysis of published empirical research has concluded that most published research findings are false.[3] Another review of empirical evidence on peer review concluded:
At present, little empirical evidence is available to support the use of editorial peer review as a mechanism to ensure quality of biomedical research. However, the methodological problems in studying peer review are many and complex. At present, the absence of evidence on efficacy and effectiveness cannot be interpreted as evidence of their absence.[4]
Surely a central aspect of the methodological problems is that peer-reviewed articles are the primary scholarly assets of scholars today. If persons were to recognize that editorial peer review has no fundamental value, the scholarly economy might collapse.
In the communicative circumstances of the Internet, publish everything is a good fundamental principle. Classical peer review tends to be understood as a value-creating quality filter for publication. It is associated with competition to get as many peer-reviewed publications as possible. An alternative to peer review is open-access competition for attention and approval: “publish everything and then let the world decide what is important.”[5] The Internet allows a huge amount of information to be made available to a huge number of persons at much lower cost than would be possible with print publications. That technological transformation fundamentally favors making any symbolic work available to everyone.
However, making any symbolic work available to everyone is not sufficient for a secure, well-functioning scholarly economy. The effect of social influence on symbolic choices isn’t well understood. Yet one can reasonably believe that fashions and celebrities are not manifestations of a good process for seeking truth, although they may be an inevitable aspect of any real social process. “Let the world decide what is important” ignores the reality and importance of human social nature and human institutions. A good social structures for symbolic competition in truth-seeking is a key challenge for the new Internet world.
* * * * *
Update: more on reforming peer review
Notes:
[1] Douglas P. Peters and Stephen J. Ceci (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5, pp 187-195 doi:10.1017/S0140525X00011183
[2] Richard Smith (2010). Classical peer review: an empty gun. Breast Cancer Research, 12(Suppl 4):S13 doi:10.1186/bcr2742
[3] John P. A. Ioannidis (2005). Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
[4] Jefferson T, Rudin M, Brodney Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database of Systematic Reviews 2007, Issue 2. Art. No.: MR000016. DOI: 10.1002/14651858.MR000016.pub3
[5] Smith (2010), cited above, p. 3.