Reviewing the Reviewers: Training Neural Networks to Read Peer Review Reports

The study of academic peer review is often difficult owing to the confidentiality of reports. Asan occluded genre of writing that nonetheless underpins scientific publication, relatively littleis known about the ways that academics write and behave, at scale, in their reviewing prac-tices. In this chapter, we describe for the first time the database of peer review reports at PLOSONE, the largest scientific journal in the world, to which we had unique access. Specifically,we detail the approach that we took to training a multi-label, multi-class text classifier usingthe TenCent NeuralClassifier toolkit to examine the peer review reports. Although this resultedin a predictable failure to produce accurate levels of recall and precision, we argue that as thesetechnologies further develop there are a range of uses – for both good and ill – that could be usedto machine-read these archives

• https://www.cambridge.org/core/elements/...