How Bad Is Reviewer 2, Actually? Data from a Philosophy Journal
Does Reviewer 2 deserve that terrible reputation?
As Jonathan Weisberg (Toronto) notes:
Reviewer 2 is accused of a lot. It’s not just that their overall take is more severe; they also tend to miss the point. They’re irresponsible and superficial in their reading. And to the extent they do appreciate the author’s point, their objections are poorly thought out. What’s more, if they bother to demand revisions, their demands are unreasonable.
Do the complaints stand up? Professor Weisberg is one of the managing editors at Ergo, and he looks at some of that journal’s data to answer this question.
First, he looks at whether there is any connection between the order in which referees are invited and their severity. It turns out the answer is no. Then he looks at whether the speed with which reviewers complete their reviews affect the severity of their reviews. Here there is an effect, but not one that matches up with the reputation: referees who take longer are less severe in their refereeing.
For further analysis, see Weisberg’s post here.
What about the order in which reviews are reported to authors? My impression is that editors prefer to hand out the good news first, at least when the decision is an R&R. That is, reviewer number as seen by the author isn’t independent of content. Of course, that applies only to journals that send individual decision letters with reviews copy&pasted in, but that’s still a lot of journals.
Ergo relays reports to authors in two ways.
If you’re following your submission’s progress on the website, reports become visible as they come in. (Although there’s a seven day delay to allow for vetting, in case there’s anything inappropriate or in violation of the journal’s policies.)
If you prefer to just wait for the email with the editor’s decision, the reports are included in the order the referees were invited.
My impression is that the second method is pretty standard. It was baked into the editorial management software Ergo used to use (the Open Journal System). And I suspect it’s also baked into commonly used commercial systems like Manuscript Central. But I haven’t worked with those systems as an editor, so I can’t be sure.
Doesn’t the Ergo system also identify reviewers by a sequential letter based on the order in which they were invited? I seem to remember managing a paper a while back in which the author eventually god comments from reviewer D and reviewer H.
Yes, that’s right, same as the Open Journal System.
I don’t think that the ‘Reviewer 2’ meme was meant as a sincere scientific hypothesis.
Oh no, you mean the grinch stole #reviewer2 🙁
Thanks for the feedback, Reviewer 2! You’re entirely correct, of course: one of us has missed a joke here.
If more serious fare is what you’re after, you might be interested to read the previous post on author gender: http://jonathanweisberg.org/post/Author%20Gender/