Should You Referee the Same Paper Twice, for Different Journals? (guest post by Eric Schwitzgebel)


The following is a guest post* by Eric Schwitzgebel, professor of philosophy at the University of California, Riverside, and blogger at The Splintered Mind.


Should You Referee the Same Paper Twice, for Different Journals?
by Eric Schwitzgebel

Uh-oh, it happened again. That paper I refereed for Journal X a few months ago—it’s back in my inbox. Journal X rejected it, and now Journal Y wants to know what I think.  Would I be willing to referee it for Journal Y?

In the past, I’ve tended to say no if I had previously recommended rejection, yes if I had previously recommended acceptance.

If I’d previously recommended rejection, I’ve tended to reason thus: I could be mistaken in my negative view. It would be a disservice both to the field in general and to the author in particular if a single stubborn referee prevented an excellent paper from being published by rejecting it again and again from different journals. If the paper really doesn’t merit publication, then another referee will presumably reach the same conclusion, and the paper will be rejected without my help.

If I’d previously recommended acceptance (or encouraging R&R), I’ve tended to just permit myself think that the other journal’s decision was probably the wrong call, and it does no harm to the field or to the author for me to serve as referee again to help this promising paper find the home it deserves.

I’ve begun to wonder whether I should just generally refuse to referee the same paper more than once for different journals, even in positive cases. Maybe if everyone followed my policy, that would overall tend to harm the field by skewing the referee pool too much toward the positive side?

I could also imagine arguments—though I’m not as tempted by them—that it’s fine to reject the same paper multiple times from different journals. After all, it’s hard for journals to find expert referees, and if you’re confident in your opinion, you might as well share it widely and save everyone’s time.

I’d be curious to hear about others’ practices, and their reasons for and against.

(Let’s assume that anonymity isn’t an issue, having been maintained throughout the process.)


art: Andy Warhol, “Shadows”

guest
41 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
chiulynnc
chiulynnc
4 years ago

This happened to me recently. I asked the previous journal editor for details about the circumstances of the paper (they withdrew with a R&R).
The editor actually told me my comments were greatly appreciated. I then disclosed my previous experience with the next journal. The editor told me to give the paper a brief look– if it’s not substantially different, they will accept my previous comments. If it is substantially different, they would appreciate it if I could review the paper again. Turned out the paper was significantly different and I liked what the authors did. Given the circumstance (withdrawal instead of rejection), blessings from both editors, and the significant changes in the paper, I thought it was a good idea to re-review it. Though at this point I felt like I was one of the readers of the manuscript… having gone through at least three rounds of edits with it throughout the year.Report

Sad Eyed Philosopher of the Lowlands
Sad Eyed Philosopher of the Lowlands
Reply to  chiulynnc
4 years ago

chiulynnc’s actions seem like the right ones to me. Perhaps I’m in the minority, but I tend to treat a rejection like an extreme form of R&R. Unless I receive a desk rejection, I never send the same, rejected paper to a different journal. So I would be disappointed if a referee were to refuse reviewing a submission of mine just because s/he had reviewed it before.Report

sara
sara

thumbs up for your username :^)Report

Elizabeth Harman
Elizabeth Harman
4 years ago

I think it’s wrong to referee the same paper twice, either recommending rejection each time or writing reviews that are sufficiently negative to lead to its being rejected. One shouldn’t take such a big role in messing up someone else’s career. Philosophical tastes vary widely, and this author’s career should not be held hostage to one person’s tastes.Report

Hayze
Hayze
Reply to  Elizabeth Harman
4 years ago

Though one would hope “tastes” don’t have that big a role in assessing other people’s work.Report

ejr
ejr
Reply to  Hayze
4 years ago

One would hope that. But that’s not the way our world works.Report

krell_154
krell_154
Reply to  Hayze
4 years ago

Some would say that philosophical intuitions are a matter of taste, in a sense (I disagree). But one way or the other, different philosophers might have different views about the quality of some work, due to criteria that are very hard to explicate. I think that’s what the parent comment was aiming at.Report

Michael Brent
4 years ago

This happened to me as well. A few months after recommending rejection of a paper, I was asked to referee one with the same title as before, but at a different journal. Before accepting the new referee request, I asked the editor to send me a copy of paper so that I could read it over with the prior version open beside it, as well as my original referee report. As soon as I realised that substantial revisions were made to the paper, I agreed to referee the new (and much improved) version, and eventually recommended that it be published, subject to a few very minor revisions.Report

Sara L. Uckelman
4 years ago

I think it’s going to depend on the reasons for rejection. Scope and fit are certainly a reason why a referee might recommend rejection for one journal but acceptance for another.Report

Kaave Lajevardi
Kaave Lajevardi
4 years ago

I think the answer depends, among other things, on where the journals X and Y stand in the hierarchy of journals as commonly understood. A paper not good enough for the Philosophical Review might be quite good for a less prestigious journal. One does not referee a submission tout court; rather, one referees it for *that particular journal*.Report

John Schwenkler
4 years ago

This happens to me a lot. I always just ask the editor how they want me to proceed. Rationale: Any costs of having a paper refereed twice by the same person need to be weighed against the costs of having the editor work to locate another willing referee.Report

John Schwenkler
Reply to  John Schwenkler
4 years ago

(PS. In my experience, editors almost always say to go ahead and referee the paper anyway.)Report

Mike Titelbaum
Mike Titelbaum
Reply to  John Schwenkler
4 years ago

I also always ask the editor (at the journal that sent the second request), and I also typically find editors are fine with my re-reviewing. That is, that’s my practice when the paper has been altered in-between. When I get the same word-for-word paper sent to me, I get annoyed.Report

Mike Titelbaum
Mike Titelbaum
Reply to  Mike Titelbaum
4 years ago

(Having read more comments below, I should qualify that I wouldn’t be annoyed at receiving the same word-for-word paper if I had rejected it from the earlier journal for fit reasons. But that’s not what’s usually going on.…)Report

JDF
JDF
Reply to  John Schwenkler
4 years ago

I think that the editor is not in a position to make a good decision here. As Eric points out in the original post, one reason to not review an essay a second time is that philosophers disagree about which issues are important, which arguments convincing, and so on. A reviewer might find a particular objection compelling, a particular response unconvincing, on grounds that themselves are the subject of further disagreement far beyond the scope of the paper under review. I’m sure everyone can think of a case where one of their essays has been (perhaps very confidently) rejected for reasons that seem to stem much more from the background assumptions of the reviewer than from anything in the essay, and you cannot discuss and diagnose every possible mistaken background assumption. Yet if it happens to me and you and everyone we know, there is at least some chance we ourselves are guilty of it sometimes when we review things. If so, we should probably bow out the second time so that the paper might get a look by eyes more friendly to their project on grounds of fairness. Which respect to those grounds, the ‘costs’ of having the editor local another willing referee seem pretty much irrelevant. The editor, of course, knows none of these facts, if they are facts in any specific case. Their opinion thus does not seem all that relevant.Report

Steven Hales
4 years ago

This has happened to me a couple of times. Recently I rejected a paper, and provided the author with more than 1000 words of major criticisms. Within two days another journal asked me to referee the same paper. There is no possible way that the author overhauled the thing in that amount of time. I declined, telling the editor that I had already reviewed the paper for another journal and that the author should have a fresh pair of eyes look at it. So I am in agreement with Elizabeth Harman.Report

Franz Huber
4 years ago

I am – strongly – in favor of refereeing the same paper over and over again. Otherwise a paper is sent out until it finally finds a referee who does not reject it.Report

Recent grad
Recent grad
Reply to  Franz Huber
4 years ago

If that’s true, isn’t it true whether or not you agree to referee it a second time? (Unless the topic is so specialized that there is only a small pool of referees, all of which share your stance.)Report

Dale Miller
Reply to  Franz Huber
4 years ago

This suggests a sense of mission about keeping poor papers from being published that I’ve never shared. My attitude is that if the paper really suffers from clear and serious defects then other reviewers will probably also recommend rejection. If they don’t, then maybe I was wrong. And if I was right and a weak paper is published, the sky won’t fall. Lots of weak papers are published and will be no matter what policy we all follow about reviewing the same paper for multiple journals. Frankly, when I’m invited to review a paper for a second journal I’m generally happy to have an excuse to say no, as opposed to being glad to have an opportunity to keep a weak paper from being accepted. I would feel bound to read the paper through carefully from start to finish in case changes had been made, and if I didn’t enjoy reading it before I’d probably enjoy the second or third reading even less.Report

Brian Weatherson
4 years ago

I think the arguments against re-refereeing are assuming that editors are not as thoughtful as they actually are. Editors won’t just take referees’ word for it on a yes/no; they (at least in good cases!) read the review. If the reason for a rejection is relatively subjective, then it makes sense to have another look at it. If the reason is more objective, then it might not make more sense. And the subjective/objective boundary around here is pretty vague, but there are clear cases.

For example, imagine the following example. (This is fictional, and a little more extreme than anything I’ve had, but not a lot more extreme.) I get a paper to review on Lewis. It argues against Lewis’s theory that p. But while it is possible to read Lewis’s early paper on the topic as arguing that p, Lewis makes clear in later work (perhaps in the appendix in the collected papers) that this is not the right interpretation. So I reject, since the author is just mistaken about the target of their attack. And I spell out the textual evidence – i.e., Lewis saying “I didn’t mean that p”. I then get an unrevised version of the paper for another journal three days later. What should I do? At the very least, I should tell the editors of the textual evidence that this author is mistaken.

And that’s the general case I think. If you’ve refereed an article before, I think it’s fine to say that, and send along the old report. Sometimes I reject papers because I think they are super-narrow and uninteresting, and it feels unfair to use the same referee in multiple cases there. (And the report will make that clear.) And sometimes I will reject because I think there’s an argumentative response that is easy enough to see that it isn’t worth having the argument and response in the literature. Again, that’s something that it might be worth having more referees consider – though there’s nothing wrong with letting the editors know what that response is. But sometimes I reject papers because they make mistakes about verifiable matters. And then I don’t see why another referee is needed. What’s common to all those cases is that the editors would benefit from seeing the prior referee report.Report

Recent grad
Recent grad
Reply to  Brian Weatherson
4 years ago

If only! My experience, especially recently, is that an editor will often side with the lowest referee recommendation even if it is not even close to as detailed and objective as the more positive report. I’ve had editors at prominent journals side with referees whose reports are just a few sentences, or which ask for incompatible things within the space of a single paragraph, or which reveal a clear lack of reading comprehension. This doesn’t always happen, but it happens a lot more than you’d think.Report

David Wallace
David Wallace
4 years ago

My policy is always to let the editors know I’ve seen it before, write a sentence or two summarizing my report, offer to share that report, and ask them how to proceed.

I think the “sentence or two summarizing” is important because the reasons for rejecting a paper are as important as the decision itself for this question. At least in my field, papers are often flat wrong on technical grounds, and finding expert referees is often difficult. “This paper is of insufficient interest and clarity to justify publication in this journal” is a judgement call that perhaps ought to be made independently in different journals. “This paper only works if non-abelian gauge theories have linear dynamics, and they don’t” is a different matter. Of course there are going to be intermediate cases, but I think the journal editors are going to be in the best place to assess them.Report

Matthew Smith
Matthew Smith
4 years ago

Absolutely not. Liz Harman gives the most significant reasons why, namely, that one person should not have such an outsized role in anyone’s career.

Another reason: journals have different standards. If Phil Imprint asks me to referee Paper A, that involves using one standard in assessing A. Suppose I reject A. Two days later, I receive a request from lesser-journal-X asks to referee A. Do I send my rejection report to X with a note that given X’s lower standards, A should be accepted? Or do I rewrite the report entirely but with different standards in mind? Can I really shift my standards so easily? It seems difficult.

Another reason: referees are fickle but once a person settles on an opinion, they usually are committed to it. What use is getting repetitive reports – good or bad – on a paper?Report

David Wallace
David Wallace
Reply to  Matthew Smith
4 years ago

“What use is getting repetitive reports – good or bad – on a paper?”

To the author: very little. But the author is not the primary intended audience. If the repetitive report tells two different journals “do not publish this, it’s flatly wrong on the following technical and factual grounds” then it’s serving a useful purpose.Report

Daniel
Daniel
Reply to  David Wallace
4 years ago

If the paper were so flatly wrong on technical or factual grounds, then surely another referee would pick up on this flaw. Or else the editors are doing a poor job of choosing their putatively “expert” referees (in which case the problem is something else entirely, and more systemic!). Or else the contention that it’s wrong on technical or factual grounds isn’t nearly as evident as you, merely *one* referee, might think.

I genuinely don’t see the point of one person refereeing the same paper more than once. There are plenty of experts with plenty of differing viewpoints, knowledge bases, and other potential theoretical biases. It strikes me as highly unfair to the scholarly community for one person to have such wide-ranging influence on what should be a larger project of consensus-building in the philosophical community. What may be self-evident to one scholar may not be self-evident to others. And if it really is so self-evidently flawed, then any one of the two other experts chosen to referee at a new journal (or minimally the editor) ought to be able to catch such an error before greenlighting the paper for publication.

Besides, how often are we really talking about such clear-cut factual errors? More likely we are talking about cases where something is clear-cut to someone with the ‘right’ background assumptions, but not so clear-cut to someone with the ‘wrong’ ones.

I may respect David Wallace (or some other highly-esteemed philosopher), but I still don’t think he (or she) ought to be the sole determiner of whether a given article gets published or not *full stop*. Maybe the paper doesn’t actually require it to be the case that “non-abelian gauge theories have linear dynamics” (as proposed in his hypothetical example above), but for whatever reason David Wallace just had an off-day and didn’t see clearly that this wasn’t in fact a required presupposition of the overall argument? No longer is this obviously a case of clear-cut factual error. Now it’s a more substantive question of what’s needed for the argument to go through. Another set of eyes is still useful to confirm or disconfirm this.Report

David Wallace
David Wallace
Reply to  Daniel
4 years ago

I think you may be underestimating how difficult it can be to find willing and expert referees in specialized areas.

(Also, if I say that something is straightforwardly factually wrong and this isn’t true, that’s what appeal processes are for.)Report

David Wallace
David Wallace
Reply to  David Wallace
4 years ago

(I should have added: appeals processes, and the assessment of the other referee.)Report

Daniel
Daniel
Reply to  David Wallace
4 years ago

This may well be true for *certain* highly specialized fields. I can’t speak to the difficulty of finding experts willing to review philosophy of physics articles (or philosophy of science articles more generally), so I will have to take your (expert) word on it. But then I don’t think your case is representative of the usual cases of this sort of double-reviewing phenomenon. And the usual cases are, to my mind, quite clearly what we should be focused on when coming up with best practices for peer review.Report

David Wallace
David Wallace
Reply to  Daniel
4 years ago

I’d rather say that best practice should vary in different bits of the discipline, rather than try to achieve one-size-fits-all.Report

Kenny Easwaran
Reply to  Daniel
3 years ago

I think you may be underestimating how difficult it is to find willing and expert referees in *any* area of philosophy. Even in fields where there are a large number of philosophers that would be willing to referee, an editor often has a hard time *finding* those people, because editors don’t have a big list of everyone who works on the topic of this particular article, and it can often take three or four (and sometimes far more) tries to find someone willing to referee something.Report

Eric Schwitzgebel
Eric Schwitzgebel
4 years ago

Interesting conversation! So far, I’m not seeing much agreement with my asymmetrical approach.

On my Facebook page one person suggested that in re-refereeing one might be too emotionally attached to seeing that the authors have revised the paper in response to your previous comments, which might bias your judgment even if the paper is changed in other ways or a better fit for the new venue.Report

Dale Miller
Reply to  Eric Schwitzgebel
4 years ago

My policy is the same as yours, with the qualifier that I have in fact never been asked to re-review a paper for which I recommended acceptance. So my commitment to that part of the policy is only in principle at this point. I vaguely remember getting one for which I had recommended an R&R, and thinking about whether I should tell the editor that I had seen the paper before but had not recommended that it be rejected. I think I may have ended up just doing that one again, though.Report

Sam Duncan
Sam Duncan
4 years ago

I do think that one should refuse to referee a paper one had recommended the journal reject. I’m less sure about the ones where one recommended an accept or an R and R, but I lean toward saying that one should do it in that case. My reasoning is this: I get very few truly terrible papers to referee; I can only assume that the journal editors desk reject those. Honestly, most of the time when I’ve recommended rejection recently it’s usually been because I think the author’s point is trivial or that it overlaps a great deal with a point someone else has already made or even that it does so to the extent of reinventing the wheel. The thing is in cases like that I’m not always entirely confident that I haven’t just missed what makes the paper exciting and new or that my own judgments about what’s important and what’s not might not be idiosyncratic. We all make errors in this regard. I can think of more than one case where I’ve went back and looked at a published paper I didn’t think was all that interesting on first read and found it much, much more interesting when I was forced to reread it very carefully. If I know I can make errors on published work I know I can make it on papers I’m refereeing and I feel that a new and unbiased referee would be more likely to see what makes it interesting on a first read than I would on a second. (And let’s face it if there are no revisions or only very minor one’s I’m probably not going to do a terribly careful second read.)
On the other hand, I’ve only rarely went back and concluded that a paper I really liked on the first read was truly bad or uninteresting. I’ve often downgraded my estimate from “great” to merely “very good” as I’ve seen more issues, but I haven’t really fundamentally changed my judgment that something was a worthwhile paper in a long time. Granted some of the stuff I was really enthusiastic about in grad school, I now think is pretty trivial or confused, but that hasn’t happened at all that I remember for anything I’ve read since I got out of grad school. I have some doubts on this, but I’d lean strongly toward re-refereeing papers that I was initially positive about. (The issue hasn’t come up for me yet.)Report

David Wallace
David Wallace
Reply to  Sam Duncan
4 years ago

“My reasoning is this: I get very few truly terrible papers to referee; I can only assume that the journal editors desk reject those. ”

At least 80% of the papers I get to referee are either “terrible” or at any rate fatally flawed. So this supports the suggestion earlier in the thread that things look different in different areas of the field (and so looking for a uniform “never re-review” or “always re-review” policy is probably a mistake).Report

some person or other
some person or other
Reply to  David Wallace
4 years ago

David: it’s also, of course, possible (and compossible with this explanation, if each is doing some work) that your and SD’s idea of what is a “terrible” or fatally flawed paper is different. Doesn’t this possibility (which we can’t rule out) incline you at all towards thinking maybe it’s not a good idea for people to referee the same paper multiple times? I think it should. (I worry that your higher-order judgment about your own reliability as a referee compared to other people would otherwise be pretty problematic.) For what it’s worth, I’d also say that about 80% of the papers I’ve been sent to referee have been either terrible or fatally flawed; but if I encounter someone else who thinks that the majority of the papers they get sent are not like this, I’m much more inclined to think “ah, peer disagreement, what should we do, maybe I should revise my judgment of my own reliability as a referee” than “well, this must be because the papers that make it past editors in my subfield are worse than those in his subfield” or “this person must have had better luck than me with respect to what papers he got sent”.Report

David Wallace
David Wallace
Reply to  some person or other
4 years ago

This is a reasonable point that I’ve been reflecting on… but, on reflection, no: it doesn’t shift my views. The reason is that I don’t think there’s even a prima facie reason to think that different specialist journals in different areas of the subject should have the same fraction of submissions that are fatally flawed. Indeed, there are good reasons to expect the opposite, at least in the case of interest here: the bulk of papers I reject are because of unfixable errors of *physics*, and conversely, I quite rarely write a report that says, “this is accurate, original in the sense that this particular thing has not been done before, but of insufficient interest, so don’t publish it”. In many areas of philosophy there is no analog to “unfixable errors of physics”, and anecdotally a lot of referee reports in other areas do have the “accurate, original, but insufficiently interesting” form.

The other sanity check is one’s co-referee. Nearly all the papers I referee for philosophy journals have two referees, and you generally get to see the other referee’s report. If I was significantly more negative than my co-referee on average, I might be concerned, but I don’t seem to be.

(In writing this it occurs to me that a respectable fraction (25%?) of my refereeing is of foundational papers sent to physics journals, and I have nearly a 100% rejection rate there – I assume probably because I get sent things that the editor thinks are probably crazy but not so transparently crazy as to justify desk rejection. So my 80% figure is probably a bit high if we’re only thinking about philosophy refereeing.)Report

Jack Woods
Jack Woods
4 years ago

Again, just to remind people above, some journals don’t send reports. I’ve gotten nasty acknowledged second reports complaining that I didn’t revise the paper in light of their comments. But, of course, I hadn’t seen the comments because the first journal didn’t send them on.Report

Jack Woods
Jack Woods
4 years ago

I have to say I’m also stunned by the amount of editorializing people are comfortable with when refereeing. I tend to avoid this since it seems to me to be the editor’s job to decide what’s a good fit (and often when I get this kind of editorializing, it seems perverse, so I worry my opinions themselves aren’t standard.) Similarly with ”this paper isn’t good enough for THIS journal”; let the typical quality of the papers published determine how good a journal is instead of whatever ranking from a number of years ago is sitting around in a referee’s noggin’Report

Wesley Buckwalter
Wesley Buckwalter
4 years ago

No, people should not be doing this.Report

Heraclitus
Heraclitus
4 years ago

It is not possible to review the same paper twice.Report

Michael Huemer
Michael Huemer
4 years ago

If I’m confident in my original judgment, then I’ll referee the paper again. If I’m unsure, I’ll decline so that someone else can weigh in and presumably make the assessment more reliable.

Some things to keep in mind in thinking about this sort of thing:
– The purpose of journals and of journal referees is not to benefit authors. It is to benefit the profession by publishing intellectually valuable work and excluding less valuable work.
– Journal space is limited and relatively fixed. Thus, getting a paper published should be expected, on average, to cause one other paper to not get published that otherwise would have been. Because of this, there is no reason to have practices that are biased in favor of publication, rather than against.
– The burden of refereeing is very heavy on the profession. It does not make sense that one and the same paper should take up the time of, say, a dozen different people. It makes sense to me that if someone has already done the work to review the paper once, the profession should save time by letting that same person review it again.Report