The Questions a Referee Should Ask of the Paper They’re Reviewing
Concerned that referees for philosophy journals too often recommend a paper be rejected merely because they can think up objections to it, Richard Yetter Chappell (Miami) suggests a set of questions they should focus on instead.
They are:
(1) What (if anything) is interesting and original about this paper? How interesting is it? (How much light does it shed on previously-shadowy areas of logical space? How philosophically important is the illuminated content?)
(2) Are there any egregious errors or oversights that would need to be addressed before the paper was potentially publishable?
(3a) How cogent are the paper’s central arguments?
(3b) Do you expect most other experts would share your verdict, or is there significant room for reasonable disagreement here?
(3c) How likely is it that a reasonable, well-informed (but currently agnostic) spectator of the debate would shift their credences as a result of reading this paper?
(4) Compared to other papers recently published in this journal, is this paper (vastly or slightly) more or less likely to be (a) widely cited, (b) assigned in graduate seminars, (c) significantly reshape the debate, and (d) still read or remembered a century hence?
He adds:
With some wonderful exceptions, referee reports in philosophy tend to systematically overweight (often idiosyncratic) judgments about whether the argument is successful (3a). Questions like 3b & 3c strike me as much better for purposes of objective professional evaluation. And questions (1) and (4) strike me as most important to the overall evaluation that a paper warrants.
Note that scoring well by the criteria in (4) seems sufficient for warranting publication in a top journal. Whatever complaints a referee might have about a paper, they obviously shouldn’t deprive the discipline of a paper that could prove so significant and widely valued. Those are clearly the papers that journals should want the most.
You can read his full comments here.
Related:
“How to Write a Referee Report”
“Advice on Refereeing Papers“
I have rejected virtually everything I have refereed, with n in the mid double digits somewhere. I have (to my mind) never rejected a paper on the grounds that I disagree with it.
Instead, every single paper I have rejected I have rejected because 1) it fails to engage the existing literature (and in many cases is thus a recreation of an already-recalled wheel), 2) it fails to address obvious objections (such that readers are likely to leave the paper frustrated rather than informed), or 3) it is disorganized and I have to do too much work to extract the argument.
Moreover, while I have had papers rejected, I don’t think I have had papers rejected on grounds 3. Instead, they get rejected for similar reasons. (Of course, sometimes I have felt that the referee was wrong in their judgments as to those reasons, but still!)
Maybe I’m just (un)lucky? But it seems to me that the problem is not overly persnickety referees but submission of half-formed papers (and the practical pressures that lead to those submissions).
If you have reviewed 50 or so papers and rejected virtually all of them then perhaps you should seriously consider that you are one of the people who should reform their practices.
More specifically, I suspect that papers that you view as ‘fail[ing] to address obvious objections’ might not look that way to others and you tend to, in Richard’s words “systematically overweight (often idiosyncratic) judgments about whether the argument is successful”.
Or at least, there is some some statistical evidence in that direction.
Oh, absolutely! I have no doubt that I am fallible. Hence, I regularly discuss rejections with colleagues, and I read other referees’ reports when they are made available. But those attempts to gather other data, and my own attempts to look for the creative and the new in the manuscripts I am sent have regularly confirmed my pre-existing sense of the papers. And it isn’t the case that I’ve only recommended publication where I’ve agreed with the papers’ conclusions (in fact, in most cases where I’ve recommended publication, I’ve _disagreed_ substantially in some way).
From my perspective, from the perspective of my colleagues, from the perspective of the other referees whose verdicts I see, it sure looks like the rejection rates we see at journals (which are consistent with own refereeing) are right about where they should be. And that story is consistent with recognizing the huge pressure to publish.
So, other than me feeling like a jerk, I’m not sure what to do. I’m already putting positive effort into looking for novel arguments and trying to reconstruct charitably. I’m already sitting on rejections for a night or so before submitting and re-reading the submissions first. I could stop refereeing? I’m not being sarcastic here—the practical solution is unclear, especially if there is a real possibility that my verdicts track “Will this be valuable for someone else in the field to read if they are in this debate?”.
East Coaster — This cracks me up to the point where I’m not going to comment. Oh, Philosophers…
Thanks you, Richard, for your post. I think it’s great. I’ve refereed over 100 papers in the past ten years or so. Having done a few editorial roles here and there, I know how hard it is to find someone to agree to referee. Unfortunately, the person who eventually says yes often seems to be the person with an ax to grind.
So, I nearly always agree to referee when asked. My approach when refereeing a paper is really simple: rather than looking for reasons to reject a paper (a perspective that I suspect many people imbibe in grad school and continue to hold throughout their professional career), I look for reasons to accept a paper. I grant that this may seem overly simplistic to some people (and I really can’t wait to hear what referee #2 says!), but really this is how I view it. As a result, I recommend revise & resubmit or conditional acceptance a lot. While reading any paper I referee, I ask myself how my comments can improve the paper to make it better.
Call me naive, but I think that viewing the peer review process as collaborative (rather than as a “gate keeper” or “preventing rubbish from being published”) is a better way to frame the activity.
What a performative contradiction:
“I view the process as collaborative, so when someone responds earnestly, in part asking for help, I make sure to include a rude and exclusionary remark, especially if it is unnecessary to making my main point, and doubly especially if I have incomplete information.”
I think the point though is that you emphasized two factors in your high percentage of rejections: not “disagreement with conclusions” but failure to address literature that you think anticipated the view and failure to address an obvious objection. But these two criteria are part of what is being criticized by the post. They are your assessment that the argument is weak because it is too obviously valid (“I’m sure someone else has already say this”) or too obviously invalid (“I’m sure this glaring objection will seem damning to every other reader”). Your personal sense of the weakness of the argument as obviously valid or obviously invalid arguably stands in the way of you considering whether the piece nevertheless offers an interesting perspective, contains engaging insights, or makes a thought-provoking intervention that could promote further discussion. Does this seem like a possibility?
I think I have lost the dialectic. Obviously I could be refereeing incorrectly, and obviously I could be making mistakes and overlooking things!
That’s why I acknowledged my fallibility, and that is why I non-sarcastically asked for advice!
That is also why I: 1) regularly discuss rejections with colleagues, 2) read other referees’ reports when they are made available, 3) try to check that mere disagreement is not driving my overall verdicts, 4) try to look for novel arguments, 5) put work into charitable reconstruction, 6) sit on rejections before submitting them to see if my mind changes in the light of a new day, 7) re-read submissions after drafting my comments and before submitting them.
As I wrote before, “I’m not being sarcastic here—the practical solution is unclear [to me].”
“I regularly discuss rejections with colleagues.” Wait, what? Every journal that I am aware of states that the review process is supposed to be confidential. You should *not* be discussing papers that you reject with colleagues. The paper you are reviewing was sent to you and to you alone to evaluate yourself.
Man. Look, I don’t forward the papers to colleagues, and I don’t say, “Oh, hey, I am refereeing the paper entitled ‘X’ and here is the relevant passage.” But, because I have taught classes before, I know how to discuss an issue with someone in ways that protect privacy.
Also, for what it is worth, I just checked my most recent refereeing request. Maybe they’ve tucked a confidentiality provision into the terms and conditions somewhere, but it is in neither the invitation nor the acknowledgement of my accepting the invitation. (And, I just checked: the only reference to ‘confidential’ in the guide we get is to the comments we submit as referees.) Maybe I’m refereeing only for state-of-nature journals?
Also, I am not running together validity and invalidity with the factors you referenced.
I see my two concerns as more closely tracking the original factors 1 & 4. But ymmv.
“So, other than me feeling like a jerk, I’m not sure what to do. I’m already putting positive effort into looking for novel arguments and trying to reconstruct charitably. I’m already sitting on rejections for a night or so before submitting and re-reading the submissions first. ” One approach you could take is to disclose to each editor that you have rejected virtually every paper out of 50 you ever refereed, and then tell the editor the rough percentile quality rank of this paper compared to all the other ones that you have rejected
This is a good idea—thanks!
I very much agree with everything Chappell suggests. It honestly describes what I think I’ve always tried to do as a referee. But I also get a lot of papers exactly like you describe. The most common are the papers whose whole point has been better articulated and defended in some other work. I’m very hesitant to pull out the old “does not enage with the relevant literature” line as grounds for a flat reject, especially since a lot of judgments about what one must read that I’ve gotten myself are often quite arbitrary. But when a paper basically reinvents the wheel I feel like I don’t have another choice. My suspicion is that a lot is of these are seminar papers that got good grades but aren’t journal worthy. I understand why people send those off. I don’t blame them one bit. But if I don’t reject papers that don’t contribute anything new I’m not doing my job.
I’m always surprised when I read stories like this! I know that refereeing is distributed in systematic and weird ways (e.g., some people are constantly getting asked, and some are never asked) but the statistics you cite are so different from my experience that I’m still surprised.
In 2023, I have written 31 referee reports, and in 6 of them I recommended rejection, and there were 8 in which I recommended acceptance. (All the others were some variety of revisions.)
In 2022, I wrote 36 referee reports, and recommended rejection 7 times, and acceptance 10 times.
It’s a little bit hard to go through my spreadsheet and identify how many of these are repeat papers, and how many of the cases where I recommended revision the paper got rejected (or accepted without revision, or the author withdrew it).
I started this spreadsheet because I once thought I had observed that I was accepting the vast majority of papers I referee – but after a friend expressed disbelief, I started the spreadsheet and discovered that my overall final disposition rate is much closer to 50-50, but it’s true that the vast majority of my refereeing time is dedicated to papers that eventually get accepted.
Given that actual acceptance rates are said to be much lower than this, and there isn’t a huge amount of desk rejection at most journals, I know there must be others out there who spend a higher fraction of their refereeing time rejecting papers. But it still surprises me.
I probably reject 75% of papers I review, but I think in most cases it’s because there’s some invalidating technical error. That might be specific to some subfields.
I think at least part of what’s going on here has to be that a referee’s professional standing is correlated with the kinds of papers they get invited to review, with more senior and esteemed folks (who are perceived to have more demands on their time and perhaps less patience for fodder) getting assigned more promising and ambitious-looking submissions. Since these are better-quality on average, they end up having higher acceptance rates. That seems to match my experiences on both sides of the editorial desk.
(Not that you’re not also a shining light of humanity and benevolence, Kenny.)
Just out of curiosity, when you read the papers from the journals you review for, do you think that most of the papers published there should not have been published?
No, not generally.
I realize that was ambiguous: No, I do not generally think that most of the papers I read which are published in the journals I review for should not have been published.
Thanks for sharing this, Justin!
I should flag that it’s very hard to know how influential “objection-minimizing” standards really are, so I don’t mean to take too strong a stand on that. Some reasonable grounds for skepticism were expressed by several folks on the post’s public facebook thread. Still, as I replied there, it seems like it couldn’t hurt to make “good-seeking” standards more explicitly encouraged.
I worry that my own refereeing contributes to the problems Richard rightly identifies. But it’s not because I am committed to some picture of a good philosophy paper as being objection-proof. Rather, the issue is about what I can know, especially in cases where a paper is not squarely in my area of expertise. Identifying objections often requires little knowledge from outside the paper under review. Consequently, when I spot an objection I’m usually quite confident that my objection has merit. On the other hand, assessments about how interesting a paper is, how original it is, and so on require significant background knowledge about the literature and the field. As a result, I am usually much less confident about those judgements. I have, after all, frequently had the experience of reading a paper that I found super interesting and illuminating and then, as I pursued the topic further, realized that there were lots of other papers around saying basically the same thing. That the paper was illuminating for me turned out to depend on my ignorance, and another scholar might well justifiably have deemed the same paper completely unoriginal and uninteresting.
For reasons like that, I suspect that I have overemphasized objections in my own refereeing, thereby inadvertently contributing to effects that I also lament. I’m not sure, though, how to get me to be more confident about positive judgements about interestingness or originality.
This is a very helpful perspective. It seems to me to mirror the common experience (this is purely anecdotal) of many graduate students who are producing relatively original work (for their career stage) and who are frustrated that the bulk of their supervisor’s comments concern writing style and pointing out obvious objections they have yet to consider. No doubt much of that is merited, especially when it comes to forming graduate students (however promising they may be), but the question is whether this is part of the systemic over-weighting of this type of criticism as the original post suggests.
At the referee-submission level, as with the supervisor-student level, it is probably rare (or at least uncommon) to find a perfect fit for the hyper-specialised area of philosophy someone is working in. Referees/supervisors are busy people, but they still want to do a good, professional job, so they stick to what they know best. Originality judgements get under-prioritised.
Part of me wonders if this problem is more acute with ‘general journals’. However, I did hear once from a friend who said that in their experience it was easier to get an ethics paper published in general journal than a specialist one, presumably because in the latter there was a greater awareness of the existing literature.
I still fail to see what the incentives are for the referee to do a good job. A referee is not paid (which is the main incentive for a job), and refereeing has no recognition in the job market or in promotion (and rightly so, probably, given that there is no control on the quality of the performance).
Absent incentives or possible enforcement, suggestions to improve the practice sound like a mere wish that people will be good. A wish in tune with Christmas times, perhaps.
I think peer review in the humanities is clearly and essentially flawed. I don’t suppose there are many other options, but because there are almost no rewards, because what we write & think about does not typically possess qualities of facticity, because disciplines by their very nature keep gates, because disciplines are tribal & faddish, and because many reviewers seem to feel put upon by having to read papers, we end up in echo chambers and with gatekeepers & ax grinders manning the ship. Certain approaches or points of view become more or less unexamined dogma, reiteration is prized over innovation, and “the books” remain more or less closed to those who or think distinctively.
Of course none of this specific to the academic humanities, but altogether it makes for a pretty nasty brew. Add in some ego, power grabs, and nepotism and here we are. Clearly we need trained readers, and clearly we need review—willy nilly is even worse.
Realistically, it seems to me that the onus to begin to address some of this falls on editors. I know they’re already overworked, but ultimately it’s the editors who need to intervene in cases of obviously ungenerous reviews, who need to make recommendations based on careful readings of reviews in split decisions, who should generally lean into r&r over rejection if there’s no strong case for rejection.
“ Absent incentives or possible enforcement, suggestions to improve the practice sound like a mere wish that people will be good. ”
Isn’t it more advice for people who want to be good (I.e., most people) as to the best way to do it?
Fair enough. I probably just disagree on “most people’, which makes the advice not so crucial.
I’m certainly not interested in reading papers whose arguments are not cogent. I hope referees don’t stop rejecting them.
This is a willfully uncharitable reading of what the post is proposing that’s truly worthy of referee 2. It’s quite clear he’s not saying referees should accept articles that commit logical fallacies or have premises that are factually false. Papers guilty of those things would fail his 3b, 3c, and 4 miserably. Instead what he obviously means is that the referees’ own belief that the author is wrong on some contentious and controversial commitment shouldn’t be enough to reject.
> Instead what he obviously means is that the referees’ own belief that the author is wrong on some contentious and controversial commitment shouldn’t be enough to reject.
That’s surely a willfully uncharitable reading of current refereeing practices.
Presumably few referees explicitly conceive of themselves as rejecting a paper “because it appeals to a contentious premise that I happen to disagree with.”
But it seems plausible enough that many are in the habit (as taught, e.g., in grad school) to hunt out possible objections to a paper as their dominant mode of evaluation. And it will naturally seem saliently objectionable to many if the paper rests on an assumption that strikes them as false.
My suggestion is that refereeing could be improved if (i) referees pause to consider the possibility that their judgment on this point could be idiosyncratic: even if *they* reasonably reject a premise, might *others* reasonably accept it (including some who don’t initially accept the conclusion)? and (ii) beyond just considering objections, also try to explicitly assess other virtues like interestingness.
Do you disagree with these suggestions?
(To be clear, my OP wrote: “Transparently bad arguments are certainly of little interest.” So I don’t disagree with the literal content of your original comment. But if you think that you’re disagreeing with me, I’d be curious to hear where you think the disagreement lies.)
The acceptance rate of journals is ridiculously low. I dont know what can be done about this. There is enormous pressure to publish. The system is obviously flawed. Maybe there needs to be a collective incentive to share one’s academic work as pre-print?
I feel like ‘previously-shadowy areas of logical space’ aspect of peer-review is not very infrequently ignored nowadays (willfully or otherwise). I see papers in very good, even even top-5, journals which are mere regurgitations of old arguments. Of course there is nothing wrong providing a novel defense or a new refutation of an old argument, but the papers in question do not very often even acknowledge the arguments they are basically restating and do not state what is novel in their argument.
In general, I feel like philosophy peer-review is very punishing of novelty and very accommodating of unoriginal and even mere restatement of old ideas (assuming everything else in the stated criteria is fullfilled).
“I feel like philosophy peer-review is very punishing of novelty and very accommodating of unoriginal and even mere restatement of old ideas (assuming everything else in the stated criteria is fullfilled)”
This is the case across the humanities, I think.
It would be interesting to know whether – and when – the ratio of ‘old hands’ v. ‘new hands’ has changed among reviewers in philosophy: how many of the reviewers are presently at the beginning of their careers, and how many are older (and wiser?). Considering the explosion in submissions in the last 20 years, I suspect that more and more younger faculty/grad students have been recruited as reviewers. This may have changed how papers are reviewed (research topic: ‘The psychology of newly appointed reviewers’). Now reviewers tend to ask authors to respond to objections (if I read the comments right) – rather than assessing the originality of the paper. If many reviewers are freshly minted philosophers, then this emphasis on objections would make sense – that’s what they have been taught to do for their PhDs. Judging originality is more difficult because you have to have a good overview of the field. The fact that we can come up with objections to Plato, Aristotle and Kant doesn’t mean that their writings should be rejected, or that they should ‘rewrite and resubmit’. Inventing hypothetical objections doesn’t make a paper better – it diminishes its readability [unless they write in the style of Aquinas]. As long as the paper is cogent – and original, let’s leave it to the readers to mount objections. That is also much more fun for them, rather than having it all ‘pre-formatted’ by reviewers and authors.
I’m not sure, but I think I reject fewer papers than I used to. But I also think that the papers I get asked to referee are better than they used to be. Obviously this might just be an illusion on my part! But here is something I wonder, and which (perhaps) only editors can answer: If an editor has a paper that’s fairly bad, is he or she likely to send it to an early-career person, and to send promising papers to more seasoned referees?
Also: Perhaps someone who rejects a lot of papers is someone who is in the stable of referees for a journal that doesn’t do much desk-rejection.