Not Refereeing the Resubmitted Paper You Recommended for Revision
Here is something that happens in the world of academic philosophy publishing:
(a) Referee agrees to referee a paper.
(b) Referee deems the paper worthy of a “revise and resubmit.”
(c) Editor endorses and conveys the R&R judgment along with the referee’s comments to the author.
(d) Author revises the paper and resubmits it.
(e) Editor asks original referee to review the resubmitted paper.
(f) Referee says no, not going to do that.
My view on this is WT(f)?
But perhaps that’s an unusual response. What is the consensus on when all of the above happens? Does (a) commit the referee to saying yes at (f)?
Justin, your view about this is right. Refusing to re-evaluate a paper that you originally gave an R&R is a total di*& move. I’ve never met anyone who was not pissed off when they’ve spent weeks making changes aimed at satisfying a (usually idiosyncratic) set of requests made by some referee, only for that paper to then not actually go back to that referee.
It has recently happened to me that I accidentally found out after the first round of review that a paper was written by a good friend. I don’t feel like I can be very objective now if the paper comes back to me, so I will ask the journal to have the R&R reviewed by someone else. But of course this is not a standard case, so I agree that one should usually be willing to look at the revised version.
I agree. But have also been asked to re-evaluate the same R and R paper maybe 5 times… There has got to be some limit.
I would think there’s a natural limit here: if you recommend a paper for R&R and it comes back to you a second time, what reason would you have to recommend another R&R? I can see 3 possibilities here:
(1) You failed in your initial review to identify some feature(s) of the paper that is a major enough deficit to warrant an R&R (rather than a conditional acceptance). In this case, we might think you have an obligation to referee the paper again given your oversight.
(2) The revised paper did not adequately address the feature(s) of the paper that were major enough deficits to warrant the initial R&R. In this case, it’s a good bet that although it might be possible that this paper could be revised satisfactorily, *this particular author* may not be capable of revising it satisfactorily. Giving it another R&R at that point is a good use of nobody’s time, and it should be rejected.
(3) The revised paper was so drastically revised that it has introduced new issues or concerns that warrant another R&R. In this case it’s less clear to me that the reviewer has an enduring obligation to review the third version, but I also have a hard time imagining a paper that was good enough to warrant an R&R initially but nevertheless require 5 revisions that were so drastic as to introduce new, serious issues at each revision.
“(2) The revised paper did not adequately address the feature(s) of the paper that were major enough deficits to warrant the initial R&R. In this case, it’s a good bet that although it might be possible that this paper could be revised satisfactorily, *this particular author* may not be capable of revising it satisfactorily. Giving it another R&R at that point is a good use of nobody’s time, and it should be rejected.”
I’d say that failure to fully address the deficits isn’t sufficient for concluding that the author is unlikely to be capable of doing so (and so rejection is warranted). At most, this is a pro tanto reason for such a conclusion. Whether rejection is appropriate will depend on many other particulars of the paper and the revision. To name just three: whether the paper, given the revision in question, would make a really, really big contribution (the stakes for drawing the negative conclusion are high); whether the author made up 90% of the deficit, but there’s a remaining 10%; and whether the author was within reason to think she had satisfactorily addressed the problem.
I think that’s right. And, there’s another possibility that referees have to keep in mind: maybe your explanation of what was wrong and what it would take to fix it just wasn’t as clear as you thought. Maybe the author misunderstood, and it’s at least partly your fault.
If you think about how often this happens in a question period after a talk, it might make you suspect it happens fairly often in referee-author exchanges, even though referee reports are presumably more carefully composed than colloquium questions.
Genuine (and perhaps naive) question. Presumably the editor’s *endorsement* of the R/R at c involves some understanding of what’s at issue, both in the paper and with respect to requested revisions. So given c, why don’t editors just make the call at d?
I’ve never worked for a journal (other than as an author/reviewer) so I’ve no idea if there are procedural blocks or more substantive concerns.
I don’t know whether this is relevant to the question, but some journals stipulate in the R&R letter that subsequent submissions will be treated as a new submission, (re)entering the review phase just like any other paper. Presumably, in these cases, there is a significant likelihood that the paper will have different reviewers anyway.
I agree with the original poster’s claim here. I once was given an R&R from a very good journal, and the main concern was that I should take ‘the Williamson stuff out’ because it was irrelevant to my argument. I then took the aforementioned Williamson stuff out and resubmitted. And then the paper was REJECTED by a different referee who was like ‘This is a good paper, but I am rejecting it because there should have been some discussion of Williamson’. I was raging mad!!!
This likely will not come as any consolation, but surely that is grounds for an appeal to the editors? “Referee B criticized me for something that I was specifically instructed to do by Referee A, how was I supposed to handle this?”
I have a feeling that a paper I reviewed once might have been in a similar situation–I got it, sent in a report suggesting a reasonably extensive R&R, and the editors wrote back saying something along the lines of “This is already an R&R, please just say whether this is OK as it stands.” Which I thought they handled well.
[OT, but something this reminded me of; has anyone got a paper rejected because you didn’t discuss your own work? What do you do when that happens?]
“[OT, but something this reminded me of; has anyone got a paper rejected because you didn’t discuss your own work? What do you do when that happens?]”
Yep. So I mentioned my own work – did the trick!
“What is the consensus on when all of the above happens? Does (a) commit the referee to saying yes at (f)?”
I’ll go out on a limb and predict that there will be no consensus. 🙂
My view: it creates a defeasible presumption of saying “yes”. Whether it is defeated in any particular case depends on the details. Absent details, it’s hard to know what to think.
Much of the above discussion seems to be premised upon an unfortunate myth: that the paper could ever go back to the “same referee” in the first place. Seriously. We are so deeply riddled with cognitive biases (including self-serving cognitive biases that lead us to deny we have many biases), and these biases are triggered to such an extent by environmental factors that can vary dramatically from ‘refereeing occasion’ to ‘refereeing occasion’. Also variable is what we’ve recently been reading, what we’ve been influenced by, etc, what we’ve eaten, what our mood is, etc. I have refereed many papers, and it would be very naive for me to suppose that–with the exception of amazing and terrible papers–I am at all consistent. Most papers aren’t either amazing or terrible, and so they go in to my ‘subject to cognitive biases zone’–which means that whether I give it an R&R, or reject, will depend largely on things like what my mood is, etc. As you are reading this, you must think I am terrible. However, I now submit a further claim which is that you are like this too.
I agree with a strong but defeasible presumption in favor of agreeing to referee if it comes back. But as John Turri says, details do matter as is illustrated by Julia’s example of learning you know who wrote it. I don’t think I’ve ever refused to re-referee a paper, but I would at the very least bring this sort of thing to the editor’s attention before agreeing to review again.
But not every re-referee request comes in response to one’s own R & R recommendation. I’ve rerefereed papers I didn’t think warranted an R & R the first time. I’ve also gone through several rounds with papers I thought were revisable so as to be publishable. I don’t think that it is unreasonable to recommend a second R & R when a paper seems like it might turn out well, but then I also think the same presumption applies as with the first R & R. I’ve been on the receiving end of such verdicts and they made my papers better.
Helping my friend move in August 2012 creates in me a certain ethos as a friend that may lead a friend to be willing to ask my help in August 2016. But it does not create in me an obligation to help them move again. And if I feel obligated against my desires, I am sure I will help badly, kicking boxes and scuffing the floor.
I have refused to review an R&R because circumstances in my life meant I would do it badly. I have refused to review an R&R because it came back so fast, I did not believe they took my feedback seriously (I requested to see the ms before I refused, and felt my suspicions confirmed).
No one is paid to review manuscripts for journals. I can see good reasons why an author might have said that “I should have reviewed the R&R.” My ethos as a good professional requires it.
I feel certain both of those ms would have been rejected had I been a good professional.
The authors had a better chance to move toward publication with a new reviewer.
Still want me to read?
Refusing to review because it came back to you too fast? This doesn’t seem fair – some people are just fast workers. And its not like those who took months to get it back were working on it for months. In fact, both persons probably took at most two afternoons. This is true whether the paper was returned in 3 days or 3 months. (You said you checked the paper first. And if so then checking the paper and seeing it was poorly done, not a fast response, justifies the refusal.
PS: Editors should feel empowered to read a review and judge whether it has met the R&R. Editors are glorified traffic controllers at some journals and we need to change that.
PPS: The Williamson in/out example makes me embarrassed for my (philosophy-adjacent) career as a prof in the humanities. This kind of idiosyncratic feedback is not what peer review is about.
I almost always re-referee unless there are special reasons not to. But I’m interested in what people think about what to do when you’re brought in as a new referee after a revision has been submitted, and after the author has responded to the initial referees. I tend to think I should be quite easy on the paper at that stage, so I often recommend acceptance when I normally would recommend R&R or even rejection (if, e.g., I didn’t really think the contribution was important enough to be in that journal).
I guess I don’t think you should be easy. But i think you should insist upon seeing the original referees’ comments and evaluate the paper on the basis of how the author has responded to them.
In my view, if you recommend R&R you should agree to review the revised version unless special circumstances (of the kinds noted above) prevent you from doing so. But here’s a trickier case: what do you do if you are asked to review a revised version of a paper for which you recommended rejection (which has happened to me a few times now)? My view is increasingly that you should refuse to do so. And my reason is that it is simply unfair to the author: you already judged that even substantial revision would not result in a publishable paper; as a result, your verdict is very likely to be rejection regardless of the changes the author has made.
I agree that you should refuse to do so, although my reason is rather different. As a *reviewer*, my attitude is that a journal is welcome not to take the free advice that I offered them, but that they should not then expect me to invest more time in offering them further free advice. As an *editor*, though, I will send papers back to reviewers who recommended reject but indicated a willingness to look at another version. Volenti non fit iniuria.
A piece I’ve co-written with someone else is up to R&R round three at a relatively prestigious journal, and it’s fairly clear that one of the reviewers in the first round did not come back to review it on second submission (and because of the new reviwer, we’ve had to revise a third time). The problem we are having is that the editor of said journal has provided us with no feedback or support whatsoever; we asked them to help provide us with some feedback about the process, as well as lodging concerns about the one reviewer who has been a constant in the process (their second review made no reference of the changes we made in the first round, and introduced completely new issues they felt we needed to address).
If reviewers change, I do think editors have a duty to advise the submitters as to what has happened, and how to deal with a potentially new reviewer requiring substantive changes. Certainly, our experience has not been good, and a little help from the editor (as opposed to complete silence on their end) would have gone a long way.