How to Write a Referee Report (guest post by John Greco)


The following is a guest post* by John Greco, who is currently Leonard and Elizabeth Eslick Chair in Philosophy at Saint Louis University, but will soon be taking up the McDevitt Chair in Philosophy at Georgetown University. It first appeared at The Philosopher’s Cocoon.


Stefan Zsaitsits, “Workshop”

How to Write a Referee Report 
by John Greco

I am sure that there will be varying opinions about how to write a referee report. In keeping with the spirit of this series, I here offer my own opinion, based on my own experience as someone who has written and read quite a few such reports. I have written them as a referee, of course. I have read some as an author, but many more as the editor of a major philosophy journal. It is this last perspective, I believe, that is most useful for present purposes. For referee reports, remember, are written primarily for journal editors.[1] That is, the defining purpose of a referee report—its raison d’etre—is to help a journal editor to make a decision about a submission. And that is what gives us insight into the criteria for a good referee report.

Already I have said something (at least one thing) that is substantive and controversial. That is, I don’t think that everyone would agree that this is indeed the primary purpose of a referee report. Many writers of such reports seem to think (or at least this is what I gather from reading their reports) that referee reports are primarily directed at authors, and for the purpose of improving the paper that is being evaluated. In my opinion, this is a mistake, and it leads to many of the problems that are typical of referee reports. In any case, below I will describe what I think are the essential elements of a good referee report, guided by my understanding that referee reports are supposed to help editors make a decision on a submission. Following that, I will go over what I take to be some common mistakes and unhelpful practices.

The elements of a good report.

In my opinion, the best referee reports have a few essential elements.

First, a good report contains a very short précis of the paper being evaluated, including

a) the thesis of the paper
b) the author’s strategy for establishing the thesis, and
c) some commentary on why the paper is important, interesting, timely, etc.
This may or may not include
d) a very broad outline of the paper.

Again, all this should be done in very short order—one or two sentences on each point. This is useful because it directs the editor to these essential elements and characteristics of the paper. But it also signals to the editor that the referee has understood and appreciated these essentials. It also demonstrates that the referee has the expertise needed to communicate them in clear and efficient fashion. In sum, a clear and efficient summary of the paper and its importance both informs the editor and builds confidence in the referee. Moreover, this kind of summary signals to the authorthat her paper has been understood and appreciated. And if there are some mistakes or misunderstandings in the report here, the author will now be in a position to refer back to them specifically in communications with the editor.

Second, a good referee report will give a clear summary of the reasons for the referee’s recommendation. That might seem too obvious to mention, but many reports fail to do this. For example, many reports will list a series of criticisms and/or virtues of the paper, but make no mention of the weight or importance of these various considerations. So, all the editor gets is a series of pros and cons, but no explanation regarding why these add up to “reject,” “accept,” “revise and resubmit,” etc. The better alternative would be to summarize what are the deciding factors among the various considerations being offered—what are the deal breakers (or deal makers) that are driving the referee’s recommendation. Put differently, a good referee report justifies the recommendation being offered, and does so in clear and efficient fashion.

Third, and in keeping with the previous theme, a good referee report will in some way flag more important vs. less important considerations as they occur. For example, a good report might be divided into the following sections: Summary; recommendation; major considerations; minor considerations; further comments for the author. This last “comments for the author” section can be used for comments that the referee thinks might be helpful to the author, but that should not factor into a recommendation for or against publication. This last point is related to my introductory remarks above, where I suggested that the primary purpose of a referee report is to help the editor make a decision. That is consistent with helpful comments directed at the author. But such comments are appropriately relegated to a section of the report, and clearly flagged as intended for such purpose.

Finally, the summary and recommendation parts of a report should be short and focused. That is, the editor should be able to understand the paper and the reasons for the recommendation fairly quickly. After that, a referee might go into detailed commentary on various virtues and vices of the paper, and might provide additional detailed comments aimed at helping the author. But all this is secondary to the guiding purpose at hand, which is to help the editor with his or her decision.

Some common mistakes and unhelpful practices.

Clearly enough, some referee reports are unhelpful because they fail to include one or another of the essential elements discussed above. I will leave it to the reader to consider the variations on how this might happen. In the remainder of the post, I will discuss two other common mistakes that referees make and that result in their reports being unhelpful. Both involve the recommendation of “revise and resubmit.”

First, referees often recommend “revise and resubmit” when they should recommend “reject.”

I won’t speculate why, but some people have a very hard time rejecting a paper outright. For example, as editor of APQ, it was not uncommon for me to receive reports that listed more than one reason sufficient for rejection, but then recommended revise and resubmit. Consider the costs of this. First, the author is now encouraged to revise a paper with the implication that the requested revisions will make the paper publishable. Second, that revised paper will now have to be refereed again, and by more people than just the referee recommending revisions. This makes more work for an already overloaded system. If there is no practical possibility that the resubmitted paper will now be publishable, the referee has done a disservice to everyone involved, author included.

Second, referees often recommend “revise and resubmit” when they should recommend “accept.”

Here I will speculate on the referee’s frame of mind. I believe that, at least in many cases, the referee has lost sight of his or her primary responsibility, which is to help the editor to decide whether the paper is worthy of publication, and not whether it could be improved in some way. What is more, in my experience as editor, recommended revisions would often have detracted from the quality of the paper, by insisting that some implausible objection be considered, for example, or by insisting that some further, tangential issue be explored. If you wonder why philosophy papers nowadays are often tedious, here is one major factor in my opinion.

Notice that in both cases (where the paper should have been rejected or should have been accepted) the referee could have included his or her comments in a “for the author” section. That is, even in a case of rejection, the referee could still suggest revisions for a possible descendant of the paper. And even in a case of acceptance, the referee could suggest revisions that she thinks would further improve an already publishable paper. But in either case the verdict of “revise and resubmit” is inappropriate, and oversteps the responsibilities of the referee.

Of course, it is ultimately the editor’s responsibility to decide whether a referee’s recommendation should be accepted, or whether a proposed revision is reasonable in that context. But the referees for a paper are usually more expert than the editor on any given paper topic, and so editors will rightfully defer to their recommendations in most cases. For that reason, referees should keep their primary responsibility clearly in mind, and write their reports accordingly. If you do, your reports will be appreciated by everyone involved.


[1] Actually, referee reports can be directed at other kinds of editor as well, such as a book series editor, or at some other person who has the role of accepting or rejecting a submission, such as members of a grant application committee. In this post, I will talk in terms of referee reports for journal submissions, but much of what I say would carry over to other kinds of referee report as well.

Related: Advice on Refereeing PapersReasons You Rejected a Paper; Deciding which Papers to Referee; An Objection Does not a Rejection Make; Should You Referee the Same Paper Twice, for Different Journals?Not Refereeing the Resubmitted Paper You Recommended for Revision; Reforming Refereeing; Should Journals Publicly Grade Submissions?

guest
18 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Kevin Harrelson
Kevin Harrelson
2 years ago

This is very informative. As a reviewer and author however I am unwilling to give this up:

“Many writers of such reports seem to think (or at least this is what I gather from reading their reports) that referee reports are primarily directed at authors, and for the purpose of improving the paper that is being evaluated.”

Reviewing and having your work reviewed are important learning experiences. It’s an element of how the work proceeds.. If you see it this way then you don’t worry about the ‘free labor’ arguments. For my part I never feel like I’m working for free for the journal, because what I’m doing in reviewing is rather an element of scholarship. Prof. Greco on the other hand writes here as if the journal editor’s perspective is the only one that matters.Report

Jonathan Weisberg
Jonathan Weisberg
2 years ago

I see lots of solid and widely applicable advice here. I’d just add that different journals have different roles, standards, and expectations. Some prioritize helping authors improve their manuscripts, for example. While others prefer quick turnaround.

A diversity of journal models is a good thing, as different authors have different needs. But it does mean that the ideal shape for a referee report will differ some from journal to journal.

Ideally journals would communicate their specific needs to referees, and referees would formulate their reports accordingly. But that’s not so much how things work in my experience. Still, everyone following the advice in the OP would be a pretty great second-best.Report

Matt
Reply to  Jonathan Weisberg
2 years ago

“Ideally journals would communicate their specific needs to referees, and referees would formulate their reports accordingly. “

Let me second this. It also would not be hard, as much of it could be semi-boiler plate that is used in the general boiler-plate emails that are sent out. By making the job of the referee easier, editors would be making their jobs easier (and, hopefully, that of writers easier as well.) The closest I’ve seen to this is probably the instructions for referees from _Philosophers’ Imprint_.Report

Jonathan Weisberg
Jonathan Weisberg
Reply to  Matt
2 years ago

I basically agree, although referees ignore the email boilerplate *a lot* in my experience.

Improving the UI of editorial management software to nudge referees towards addressing the journal’s specific needs might help a bit. But by then it may be too late. Referees often write their report before going to the system, and then they just wedge whatever they’ve written into the first box they find that’ll take it.

As long as refereeing is uncompensated, volunteer labour, journals just don’t have much leverage. Referees don’t have a lot of incentive to do as the journal asks, except insofar as they’re the conscientious type.Report

Philip Kremer
Philip Kremer
Reply to  Jonathan Weisberg
2 years ago

“A diversity of journal models is a good thing, as different authors have different needs. But it does mean that the ideal shape for a referee report will differ some from journal to journal. … Ideally journals would communicate their specific needs to referees, and referees would formulate their reports accordingly.”

Agreed entirely. I might add that journals should also communicate their expectations to authors. Thus, a journal might have something like this on their submissions page, “We ask referees to submit a report within two weeks, and we prioritize a quick turnaround over a detailed report.” Or, “We hope that referees provide helpful advice to the authors, especially when they recommend “accept” or “revise and resubmit”. Accordingly, we prioritize a detailed report over a quick turnaround and ask referees to submit a report within 16 weeks.” This way an author can have a good idea of what to expect, and won’t be justifiably irritated by a short but quick report from the first journal or a slow but detailed report from the second.Report

RJB
RJB
2 years ago

This seems rather different from what I’m used to as an editor in my field (accounting, but not very different from other business disciplines and econ). My advice would differ first in that I’d advise reviewers to *never* put their recommendation in the review that authors see, but only in the private letter to the editor. It’s the editor’s job to make a decision, and a reviewer who tells the author their recommendation just ties the editor’s hands, and if the recommendation is more positive than my decision, gives the author false hope.

Second, I want a review that, after summarizing the paper and the reviewer’s concerns, spells out the best feasible path to addressing the those concerns, without asking for what is effectively a new submission. I want a review that is very focused on improving the paper, because I see the editor’s job not as rejecting bad papers, but as publishing good ones, and most are not good enough when first submitted.

But maybe this is a difference between our fields. Most of the papers I edit are based on data gathering, and once the data are gathered the feasible path of the paper is pretty limited–throwing away the old data to gather new improved data would effectively be a new submission, so asking for that is implicitly recommending rejection. Maybe the argument-based nature of philosophy makes it harder to know what makes a revision effectively a new submission, so it’s better for the reviewer and editor just to focus on rejecting bad papers rather than improving good ones.Report

Adam Omelianchuk
2 years ago

This post was so helpful to me that I actually downloaded it for future reference. Thank you so much for writing this.Report

MrMister
MrMister
2 years ago

I agree with Greco that a paper which is hopeless should not be given a charity R&R, stringing the author along before an inevitable rejection. But, when it comes to the reason Greco gives—not making more refereeing work for an already overloaded system—it strikes me that this concern is somewhat selectively deployed. At least, I would emphasize that rejections without thorough explanations may reduce workload on an individual journal, but increase it on the system, as all that’s left for the author is just to immediately resubmit elsewhere. Framing reports as intended for the benefit of the editor in making a decision, rather than for the benefit of the author in improving this or future work, thus strikes me as likely to systematically generate lots of redundant re-refereeing work.Report

Chris Buford
Chris Buford
2 years ago

This isn’t about writing good reports, but is relevant to norms governing the editor/referee relationship. When I refereed for APQ, John informed me of the final decision regarding the submission. This was appreciated. Some review platforms allow referees to access the reports of other referees, but editors might think about reaching out to referees to inform them of the end result. I suppose this adds to the workload of editors but I found it informative.Report

Matt
Reply to  Chris Buford
2 years ago

I will second that it is very much appreciated to learn the decision on the paper, and to see the other referee report. Among the journals I’ve refereed for, _The Journal of Politics_ and _Legal Theory_ have been especially good on this.Report

Pablo
Pablo
2 years ago

This is an instructive post, and I hope may reviewers will learn its lessons. As Jonathan Weisberg pointed out, though, different journals have different editorial goals in mind. Perhaps journal editors can create templates for review that can guide reviewers to meeting its editorial goals. This would also publicize the journal’s editorial goals – does it want quick turnaround, does it want to help improve articles?

I would also like to see a post that offers guidance to editors for mediating disagreements between referees and authors. Putting authors through endless rounds of review when the author explains reasons for rejecting some of the referees’ criticisms is frustrating. Often, the new review requests that lines of thought be re-incorporated after being cut following previous reviews . At a certain point, the editor should make an editorial judgmentReport

Shen-yi Liao
2 years ago

Like Jonathan Weisberg, my meta-view is that it is good for there to be many different models of journals, including many models that I first-order disprefer.

That said, I do have a strong first-order view about the use of R&R, that is related to what Greco says about its misuses and the time it takes to do a second-round refereeing. I think an R&R should only be given if the referee lays out clearly which changes would be necessary to make the paper publishable, and those changes are clear and actionable. Roughly, as a heuristic, one should recommend an R&R if one’s comfortable NOT refereeing the manuscript again, and comfortable letting the editor decide whether the R&R requests have been satisfied. This also prevents a very frustrating phenomenon for authors, whereby referees come up with new requests after old ones have been satisfied.

While this suggestion may seem radical in philosophy, it’s actually adopted elsewhere. For example, the journal Collabra: Psychology sometimes publishes exchanges between authors, referees, and editors (a practice I also favor!), and one can see that not every R&R has to go back to referees. See, e.g., the exchange for this article: https://www.collabra.org/articles/10.1525/collabra.33/Report

Shen-yi Liao
Reply to  Shen-yi Liao
2 years ago

Addendum: Another first-order view I have is that people should—modulo usual power dynamic considerations—sign their reviews. Certainly, they should write as if they were signing their reviews. I think this somewhat goes against Greco’s idea that referees are primarily writing for editors, but ¯\_(ツ)_/¯ .Report

M
M
Reply to  Shen-yi Liao
2 years ago

I think that there is an important distinction between a conditional acceptance, which is like what you describe here, and an R&R. I think the difficulties with R&R that you describe here are really editorial failure — editors have to be willing to exercise judgment and not let the referees serve as de facto editors. Editors decide whether papers are accepted, not referees.Report

Tom
Tom
2 years ago

“This also prevents a very frustrating phenomenon for authors, whereby referees come up with new requests after old ones have been satisfied.”

Or, even worse: after having meticulously responded to all requests made by the original referee, the journal decides to send the paper to a new referee, whose requests include deleting the previously made changes.Report

Kenny Easwaran
Reply to  Tom
2 years ago

“after having meticulously responded to all requests made by the original referee, the journal decides to send the paper to a new referee”

as an editor I can say that I absolutely hate to do this, and only do it if the original referee for some reason isn’t responding to the e-mail message asking them to re-read the paper.Report

Steven French
Steven French
2 years ago

Excellent advice!
See also Beth Hannon’s ‘So now you’re a referee …’ (http://www.thebsps.org/2015/03/howtoreferee3/) which, among other things, urges referees to talk about what is wrong with the paper, rather than what they feel the author did wrong!Report

E
E
2 years ago

A. “… a good referee report will in some way flag more important vs. less important considerations as they occur. For example, a good report might be divided into the following sections: Summary; recommendation; major considerations; minor considerations; further comments for the author.”

B. “… I believe that, at least in many cases, the referee has lost sight of his or her primary responsibility, which is to help the editor to decide whether the paper is worthy of publication, and not whether it could be improved in some way. What is more, in my experience as editor, recommended revisions would often have detracted from the quality of the paper…”

This this this this this. For the love of whatever you consider love-worthy, this. I cannot see *any* possible reason not to do A. I can imagine some people will disagree with B, but if you do then at least you should agree with the following: Be careful about recommending improvements, especially if they’re (i) flagged as important considerations, or (ii) not flagged as important or unimportant (and therefore treated as important by default). Unless you’re quite confident that your recommendation would indeed be a *significant* improvement, either don’t make the recommendation at all or make sure you clearly flag it as (i) a *possible* change to consider or (ii) something of minor importance — i.e., something that should not be a deciding factor in whether to accept or reject.

We’ve all had the experiences of referees giving very unhelpful “It would be helpful if the author did X Y and Z…” suggestions. We’ve all seen the memes (https://tinyurl.com/y4ucnjzf , https://tinyurl.com/y3yfbrze , etc.). In at least many cases we should be confident that the author is intelligent and may well have thought of what we’re suggesting already and rejected it for good reason. In sum, we all have more than enough information to conclude that our own “helpful” suggestions might not be so helpful after all. So why do we all keep doing it?Report