The following is a guest post* by John Greco, who is currently Leonard and Elizabeth Eslick Chair in Philosophy at Saint Louis University, but will soon be taking up the McDevitt Chair in Philosophy at Georgetown University. It first appeared at The Philosopher’s Cocoon.
How to Write a Referee Report
by John Greco
I am sure that there will be varying opinions about how to write a referee report. In keeping with the spirit of this series, I here offer my own opinion, based on my own experience as someone who has written and read quite a few such reports. I have written them as a referee, of course. I have read some as an author, but many more as the editor of a major philosophy journal. It is this last perspective, I believe, that is most useful for present purposes. For referee reports, remember, are written primarily for journal editors. That is, the defining purpose of a referee report—its raison d’etre—is to help a journal editor to make a decision about a submission. And that is what gives us insight into the criteria for a good referee report.
Already I have said something (at least one thing) that is substantive and controversial. That is, I don’t think that everyone would agree that this is indeed the primary purpose of a referee report. Many writers of such reports seem to think (or at least this is what I gather from reading their reports) that referee reports are primarily directed at authors, and for the purpose of improving the paper that is being evaluated. In my opinion, this is a mistake, and it leads to many of the problems that are typical of referee reports. In any case, below I will describe what I think are the essential elements of a good referee report, guided by my understanding that referee reports are supposed to help editors make a decision on a submission. Following that, I will go over what I take to be some common mistakes and unhelpful practices.
The elements of a good report.
In my opinion, the best referee reports have a few essential elements.
First, a good report contains a very short précis of the paper being evaluated, including
a) the thesis of the paper
b) the author’s strategy for establishing the thesis, and
c) some commentary on why the paper is important, interesting, timely, etc.
This may or may not include
d) a very broad outline of the paper.
Again, all this should be done in very short order—one or two sentences on each point. This is useful because it directs the editor to these essential elements and characteristics of the paper. But it also signals to the editor that the referee has understood and appreciated these essentials. It also demonstrates that the referee has the expertise needed to communicate them in clear and efficient fashion. In sum, a clear and efficient summary of the paper and its importance both informs the editor and builds confidence in the referee. Moreover, this kind of summary signals to the authorthat her paper has been understood and appreciated. And if there are some mistakes or misunderstandings in the report here, the author will now be in a position to refer back to them specifically in communications with the editor.
Second, a good referee report will give a clear summary of the reasons for the referee’s recommendation. That might seem too obvious to mention, but many reports fail to do this. For example, many reports will list a series of criticisms and/or virtues of the paper, but make no mention of the weight or importance of these various considerations. So, all the editor gets is a series of pros and cons, but no explanation regarding why these add up to “reject,” “accept,” “revise and resubmit,” etc. The better alternative would be to summarize what are the deciding factors among the various considerations being offered—what are the deal breakers (or deal makers) that are driving the referee’s recommendation. Put differently, a good referee report justifies the recommendation being offered, and does so in clear and efficient fashion.
Third, and in keeping with the previous theme, a good referee report will in some way flag more important vs. less important considerations as they occur. For example, a good report might be divided into the following sections: Summary; recommendation; major considerations; minor considerations; further comments for the author. This last “comments for the author” section can be used for comments that the referee thinks might be helpful to the author, but that should not factor into a recommendation for or against publication. This last point is related to my introductory remarks above, where I suggested that the primary purpose of a referee report is to help the editor make a decision. That is consistent with helpful comments directed at the author. But such comments are appropriately relegated to a section of the report, and clearly flagged as intended for such purpose.
Finally, the summary and recommendation parts of a report should be short and focused. That is, the editor should be able to understand the paper and the reasons for the recommendation fairly quickly. After that, a referee might go into detailed commentary on various virtues and vices of the paper, and might provide additional detailed comments aimed at helping the author. But all this is secondary to the guiding purpose at hand, which is to help the editor with his or her decision.
Some common mistakes and unhelpful practices.
Clearly enough, some referee reports are unhelpful because they fail to include one or another of the essential elements discussed above. I will leave it to the reader to consider the variations on how this might happen. In the remainder of the post, I will discuss two other common mistakes that referees make and that result in their reports being unhelpful. Both involve the recommendation of “revise and resubmit.”
First, referees often recommend “revise and resubmit” when they should recommend “reject.”
I won’t speculate why, but some people have a very hard time rejecting a paper outright. For example, as editor of APQ, it was not uncommon for me to receive reports that listed more than one reason sufficient for rejection, but then recommended revise and resubmit. Consider the costs of this. First, the author is now encouraged to revise a paper with the implication that the requested revisions will make the paper publishable. Second, that revised paper will now have to be refereed again, and by more people than just the referee recommending revisions. This makes more work for an already overloaded system. If there is no practical possibility that the resubmitted paper will now be publishable, the referee has done a disservice to everyone involved, author included.
Second, referees often recommend “revise and resubmit” when they should recommend “accept.”
Here I will speculate on the referee’s frame of mind. I believe that, at least in many cases, the referee has lost sight of his or her primary responsibility, which is to help the editor to decide whether the paper is worthy of publication, and not whether it could be improved in some way. What is more, in my experience as editor, recommended revisions would often have detracted from the quality of the paper, by insisting that some implausible objection be considered, for example, or by insisting that some further, tangential issue be explored. If you wonder why philosophy papers nowadays are often tedious, here is one major factor in my opinion.
Notice that in both cases (where the paper should have been rejected or should have been accepted) the referee could have included his or her comments in a “for the author” section. That is, even in a case of rejection, the referee could still suggest revisions for a possible descendant of the paper. And even in a case of acceptance, the referee could suggest revisions that she thinks would further improve an already publishable paper. But in either case the verdict of “revise and resubmit” is inappropriate, and oversteps the responsibilities of the referee.
Of course, it is ultimately the editor’s responsibility to decide whether a referee’s recommendation should be accepted, or whether a proposed revision is reasonable in that context. But the referees for a paper are usually more expert than the editor on any given paper topic, and so editors will rightfully defer to their recommendations in most cases. For that reason, referees should keep their primary responsibility clearly in mind, and write their reports accordingly. If you do, your reports will be appreciated by everyone involved.
 Actually, referee reports can be directed at other kinds of editor as well, such as a book series editor, or at some other person who has the role of accepting or rejecting a submission, such as members of a grant application committee. In this post, I will talk in terms of referee reports for journal submissions, but much of what I say would carry over to other kinds of referee report as well.
Related: Advice on Refereeing Papers; Reasons You Rejected a Paper; Deciding which Papers to Referee; An Objection Does not a Rejection Make; Should You Referee the Same Paper Twice, for Different Journals?; Not Refereeing the Resubmitted Paper You Recommended for Revision; Reforming Refereeing; Should Journals Publicly Grade Submissions?