Little Things Journals Can Change Now To Improve the Review Process
Maybe you’re among those who are hoping for drastic changes to the practices and norms of reviewing manuscripts for publication in philosophy journals.
–
But if you thought it took too long for that journal to get back to you with a decision on your last paper, well, that was nothing compared to how long you’ll probably have to wait for the reviewing revolution.
In the meantime, are there small changes that journals can fairly easily implement to help improve the review process and the factors that contribute to its problem?
That’s the question raised in an email from a well-published assistant professor of philosophy. They wrote in requesting
a post about what small, actionable things journals can do, are doing, and should do now to improve the journal refereeing process in light of its clear and recent degradation.
For example, on more than a few recent occasions, I’ve checked in with editors for updates about my submissions at ~4 months only to learn that they haven’t even secured enough reviewers or even a single reviewer! This has only recently started happening to me.
And to make matters worse, in my view, when I inquired into whether the journals have a policy of what to do to remedy such delays, (e.g. reduce required number of referees from 2 to 1, rely instead on an editor’s judgment of a single report, etc.), I was always told that they didn’t.
The author of this email admitted that they weren’t sure that such a policy would be a good one, but they were hoping that there were some changes to policies and practices that might help make a difference. For a different example, consider the recent change at Ergo to no longer allow for a “major revisions” option for referees.
Perhaps a few relatively minor improvements implemented across many journals could add up to something major.
Having access to more referees might help, so I hope that many prospective referees volunteer their services on the new “Philosophers Available to Referee” spreadsheet, and that editors in search of referees are able to make good use of it.
What other ideas might be worth discussing or trying out?
A few related posts:
2,000 Spaces for 10,000 Papers: Why Everything Gets Rejected & Referees Are Exhausted
The “Insanely Low Acceptance Rates” of Philosophy Journals
Is Peer Review in Philosophy “Broken Beyond Reasonable Repair”?
A Modest Proposal: Slow Philosophy
The Publication Emergency
Peer Review or Perish: The Problem of Free Riders in Philosophy
The Case for a Peer Review Market
How To Alleviate the Referee Crisis: A Proposal
Pay Referees Per Mistake Caught?
A Plea for More Short Journal Publications
Why a Crowd-Sourced Peer-Review System Would Be Good for Philosophy
Should We Get Rid of Peer Review?
The Worst Reviewer/Editor Comments You’ve Received
Bad Reviewer Experiences
From the point of view of referees, I wish more journals would let me take a look at a paper before I have to say yes or no to refereeing. I rarely say no, but sometimes I might say yes to a paper that’s relatively short (say, no more than 20 pages) but would say no to a paper longer than that, just because of my time constraints. Some journals let you see the paper before agreeing, but quite a few do not. In those cases, when time is tight, I’m more likely to just say no than I would be if I could see the paper and know I have time to deal with it properly.
I think two tier refereeing systems (or their equivalent but with something like “area editors” at Ergo) are one of the ways forward here (I am a pluralist about ways forward!).
The two-tier system is something like (I think–I can no longer find a description of this on their website, but I think they used to have one, and it’s been a long time since I refereed for them but I think this is what they asked me to do) What Phil Imprint does, or what Phil Studies has recently started doing (not sure if the latter change is permanent), where they first send a paper to a referee to just ask for a relatively quick judgment about whether the paper should be put through the refereeing process (and whether they would be willing to referee it), comments optional. Then, if it gets put through the process, it looks more normal. (I assume an editor is also making a similar judgment and can overrule the referee, etc.) As a referee, I am much more likely to accept one of these “first pass judgment” requests, which takes up way less of my time, and can then lead to me either deciding that the paper is worth investing more time in or not. I suspect others are too.
Ergo basically does this same thing but with their vast army of area editors serving in the first pass role. So area editors read papers carefully and then decide whether they should be sent out for review, but the vast majority of submissions are desk rejected by them. Of course this kind of model is dangerous if there are just a few people desk rejecting everything, but I think it works well with the Ergo model, where there are scores of area editors, and multiple area editors in each subfield.
I believe Phil Review does that. For me, I don’t really have the ability to do a lighter first pass – I just automatically do a full review the first time.
At the risk of threadjacking: does anyone know what is going on with Dialectica? The editor explained in a Daily Nous post (in February) that the journal will resume putting out issues soon, but nothing has happened.
Also very curious about this…
Also, I know people are down on PPR and Nous in various ways but especially about their short-ish window of being open for submissions, but 100% of my many rejections and very few acceptances/R&Rs from them have been both relatively quick and helpful. I’ve never gotten bad comments from them. Their referees have always been super helpful and constructive (of course this might just be luck, but it also might be that they are not passing along abusive stuff or just have better luck themselves with securing responsible referees). Whatever they are doing in my opinion is working very well. One thing that I’ve noticed is that I don’t think I’ve gotten two referee reports from either place (whether it is for a rejection or an R&R), at least in the past 6 or 7 years. I’m sure some people object to this also, but I think a highly involved editor plus a single good referee is better than two mediocre referees.
And here are some other journals that, at least in my experience over the last eight or so years, routinely make publication decisions (including acceptances!) on the basis of a single referee report:
The Philosophical Review
Philosophical Studies
Mind
Analysis
Analytic Philosophy
Inquiry
Anecdotal, but I know of two people who have recent acceptances at the European Journal of Philosophy on the basis of a single referee report.
https://philosopherscocoon.typepad.com/blog/2024/01/one-paper-one-reviewer-an-easy-amelioration-of-the-referee-crisis-guest-post-by-mahmoud-jalloh.html
If editors are consistently getting turned down by referees, they should not simply ask two referees and wait for their replies. Ask three or four initially and once (if) two agree to referee (assuming you want or need two) let the others know they’re off the hook.
This should cut down on the time it takes to find referees (and of course if the referee acceptance rate is even lower they might initially ask even more). I’ve experienced granting agencies that had a definite deadline on their end do this and it seems fine.
This has happened to me as a prospective referee. I was invited and then uninvited later that day because they had received enough ‘yeses’ from others.
Yeah, we’ve started doing this at FPQ more often because so, so many people just don’t answer emails ever
1) Editors, stop using anonymous and horrible automated systems to make referee requests. Send an e-mail that can be answered with an e-mail.
2) Editors, give those requested one week to respond. If they don’t respond, send the request again. If they don’t respond, move on.
3) Editors, remind those who agree to referee when they are two weeks past their deadlines, asking for the report or a revised due date. When you sense indefinite dragging of feet, pull the plug and get a new referee.
4) Editors, tell authors the full situation with the manuscript when the (normatively) expected decision date arrives and there is no decision.
Editors, remind those who agree to referee when they are two weeks past their deadlines, asking for the report or a revised due date. When you sense indefinite dragging of feet, pull the plug and get a new referee.
I have sometimes asked for, and have been glad to receive, a reminder a week or two before the review is due. (I’m almost always on time, but often pretty busy, so the reminder is helpful. No doubt this is extra work for editors or editorial assistants, but I, at least, have been grateful for it.)
As an editor, I will never do suggestion (1). The automated system makes it clear who has been invited and when, and doesn’t get lost in my inbox in the same way that an e-mail would. It’s also really helpful for (3), since the system automatically sends some of those reminders.
I understand the value of automated systems. But it might be good for editors who complain about the amount of non-responsiveness and negative responses to wonder whether any of it is to be ascribed to the fact that journal editors send you a gumption trap to ask you to referee something, in a world in which we are barraged with impersonal surveys and other such demands for our time. Having a real live human being ask you to do something may actually help. There are lots of confounders but my experience as an editor, following this policy, was a lot more pleasant than what I’ve heard most editors describe.
I firmly believe that getting an actual email from a person would increase response rates.
Wouldn’t it be possible to do both? Send a regular email and an automated one, explaining why an automated one was sent in the regular one?
I have had this as a reviewer quite a few times. The automatic email (presumably so that the editor can use the system to keep track of everything) and then a follow up moments later from the editor saying “sorry for such an impersonal request, here’s a personal one” type thing.
Since most editors will be as burdened by day jobs and refereeing as the referees themselves, but with the added role of editing heaped on top, and keeping in mind a single paper can generate more than 20 invites, maybe referees could reciprocate the community service by making peace with the auto emails? It’s not a whole lot to ask in the scheme of things, I think.
(For what it’s worth, this is also why publishers push a lot of marketing work back onto the authors: if academics will only engage with those they consider peers, a lot of work can’t be offloaded onto those who would normally take care of it.)
Re 1, see the second “related post” listed at the end of the post.
The norm of referee reporting should be: provide just enough information for editors to make a decision (say in 1-2 paragraphs) . If it were, both refereeing and editorial decisions would take less time. It would also stymie the widespread practice of using the review process as a way of getting detailed feedback on papers that are little more than drafts. It seems to me that journals could put word limits on referee reports and guidance on what they’re looking for to effect that change.
Just to report that this would make me sad, as I think some of my papers have been vastly improved by the review process (shout out to whoever recently went through two very detailed rounds of R&Rs as my referee on one paper recently!), despite the fact that I wasn’t looking for feedback on the papers. Of course, R&Rs can also make papers worse, but I don’t know, I think they are really valuable dialectically for helping authors improve papers. And some of the most helpful ones I’ve had have involved quite a bit of feedback (and I am deeply indebted to my referees!).
I disagree with Kareem. The suggestion doesn’t allow for cases in which R&R (with extensive revisions) really is the right option. Sometimes you are asked to referee a paper with real potential that is nonetheless defective as it stands. If you have sow’s ear with the makings of a splendid silk purse, it can sometimes take hundreds or even thousands of words to explain to the author how to effect the transformation. A few brief comments won’t cut it.
As Pascal put it, “I have made this longer than usual because I have not had time to make it shorter.” If I had a word limit on my referee reports this would make them more onerous to write, rather than less onerous. I would first have to assemble my thoughts, and then fit those thoughts into the word limit (which entails picking which thoughts are most relevant, phrasing thoughts concisely, etc.). As things stand, I can write referee reports simply by writing down what I think about the paper, and I’d hate to lose that option.
I have not learned how to make a faster and shorter referee report. Maybe if there are some referees who do have that skill, it would be good to have a journal that focuses on that, and knows who the right referees are.
I am sympathetic to Kareem’s suggestion, but with a bit more flexibility. Something like: “A 1-2 paragraph report giving us the main reason(s) for your recommendation would suffice. You’re welcome to write a longer report if you think it is necessary or easier, or if you simply want to help the author, but you are neither required nor expected to do so”.
I think it would probably help for journals to give more advice to referees about what they are looking for, and to have more public discussion of what good refereeing involves. I’ve read tons of referees reports in various different roles and here are a couple of impressions I get about deficiencies in refereeing that I’ve seen that I guess most of us are somewhat familiar with, and have sometimes contributed to.
1) the tone of many reports is often poor, and too many referees come across as arrogant, dismissive, frustrated, or outraged at what they see as weaknesses in papers they referee. Referees seem to quickly forget that it’s ‘peer review’ not ‘superior review’.
2) tightly argued narrow papers tend to do better than more imaginative and speculative papers, and papers that are on broad and difficult questions in philosophy. This is one result of the tendency identified in the next comment.
3) referees are insufficiently disposed to try to find the strengths of a paper and see whether those strengths merit publication, and are overly disposed to try to identify weaknesses that they see as fatal. Of course, some papers have fatal weaknesses that rule out publication, and some reports pointing these out are well done, but many reports I’ve seen on papers by others that I think of as extremely good suggest a weakness that they find fatal that isn’t, or isn’t even a weakness. Much of the best refereeing I’ve seen where papers are rejected involves rejecting papers by identifying the strengths and evaluating them as insufficient to merit publication rather than finding fatal objections.
If I’m right that these tendencies exist, it would good for journals and the broader community to highlight them and give more advice.
Yes. I think public discussion is important partly because I don’t think we have any clearly articulated standards (in most journals) of what a referee should base their judgment on. Should an acceptable paper be one where the referee:
or what? (Non-exclusive categories there.) I don’t know that journals or the discipline in general give clear direction on this. I think 1 and 2 are certainly too strong because it is rare that a philosopher with a strongly held view is convinced by one opposing paper, or can’t think of any objections to it; but I worry sometimes that referees are working with something like 1 or 2 anyway.
Loath to jump into this discussion, because I know AJP could improve our processes and I don’t want to be seen to have tickets on ourselves. But I do think our instructions to referees are pretty clear on what we are looking for in a report. One thing we say, directly to Matt Weiner’s questions, is this:
But we invite referees to reflect on more than just the quality of argument – perhaps this is intended to be covered by Matt’s point 6:
I don’t think AJP is likely to be very distinctive among philosophy journals, except perhaps that we make our instructions to referees public? (Though I confess I haven’t looked extensively at other journals webpages.)
Suppose that Journal X requires the approval of multiple external reviewers before being willing to publish. And suppose the editor of X says that, if major revisions are made to your initial submission, they’ll reconsider whether or not to send it out to external reviewers.
In general, I dislike this practice. I’d be curious if others feel a similar way. I think that such offers, though almost always well-intentioned, typically have some problematic elements and come with some hidden costs for the author.
There is first a non-trivial chance that the editor will be requesting changes that are idiosyncratic to their own tastes or preferences, and which would not have been endorsed by the external reviewers working most actively in the area.
But second, in the event that the editor is indeed picking up on a genuine (but solvable) problem, then it is also reasonable to expect that the external reviewers, who are experts in the area, would have picked up on these genuine (but solvable) problems too in the natural course of the review process. So the review process is greatly lengthened for the same outcome.
Of course, it is the ultimately editor’s discretion as to what sorts of articles they would like to publish or not publish. But is seems to me that, if the editor has substantive worries about the article, they should either decline the article initially with an explanation of their reasons, or instead, raise their concerns in an overall appraisal they have come to make with the external reports in hand. They shouldn’t offer the mere possibility that they will reconsider whether or not to send the piece to external reviews, conditional on the author doing a great deal of work in changing their submission.
I’ve never heard of this happening, so I wonder how common it is, but I’m not sure I totally see why these are problems. I think authors need to use their judgment about whether (in this case the editor’s, but this is also relevant to just normal R&Rs, especially ones that seem fairly negative/like the chances are low) the comments will improve their paper or make it worse. If it will improve them, why not make the changes and see what happens? (And I’m also not sure it will necessary lengthen the process–if they are issues the referees would have picked up on, then you’ve killed two birds with one stone, no?) If the changes you would need to make wouldn’t improve the paper, or takes it too far from what centrally matters about it to you, you can just ignore them and move on to a different journal.
I guess the flip side of how to think about this is: the editor is giving you a choice between a desk rejection or a chance to try again after responding to their comments (that they presumably think will actually improve the paper and that they spent time on presumably because they think there is something promising in the paper–editors don’t have a lot of time!). The former seems better if you just want to move on quickly and the comments don’t seem helpful to you (i.e. you want to treat it as though you didn’t get the comments at all); the latter seems better if the comments are helpful.
Writing as an author grappling with an R-and-R, here is a proposal.
Sometimes, an author receives comments which are hard to swallow or require hard work, but which are, overall, fair or reasonable. Sometimes, however, you get back comments which seem as if the reviewer has read a different paper, or thinks all papers have to engage with a very specific literature, or which are too vague to action in any sensible way.
Dealing with the second sort of comments is nearly impossible. There is a temptation to write a cover letter saying why you haven’t restructured the entire paper around the unjustly ignored but ground-breaking work of Smith, but there’s always a sense that any reviewer who made such a recommendation will be unmoved. In turn, I don’t necessarily blame editors for passing on these sorts of comments; it can be hard for an outside party to a debate to judge these things.
So, here is a proposal: we should make it normal for authors to send back a revised version with a note asking that, and explaining why, they believe the revised paper ought not go to a particular reviewer. The editor can then decide whether it goes back to the initial reviewer or to a third reviewer or no-one else.
Would this speed up the process? No. It would potentially make it longer. It would, however, make it more satisfying.
Would this system simply lead to a lot of people saying “stupid reviewer”? Maybe, but the editor’s job is precisely to make a judgment – she might think that the author’s complaints sound like “special pleading” and send it back to the initial reviewer; there’s no veto power here.
As a sometime editor as well as an author, how do I think about this proposal? I hate it! It would make editing even harder. Still, I’m already often in a position where an author’s “response to reviewers” is, in effect, a politely veiled “what the hell is the reviewer on about?” At least this norm would make the situation clearer.
In the spirit of the post, here are some little things journals could do to help out reviewers with R&Rs.
(1) Tell reviewers in the email request to review an article on a second round of an R&R that it is an R&R and not a request to review a new article! I say this because when one is reviewing many papers, it is easy to forget that this is an R&R you asked for, and think instead that one is being asked to review a new article or (even worse) an article one previously recommended for rejection at a different journal. I know it sounds like reviewers should be able to remember this, but if one gets four or five review requests for articles with similar titles and themes, it can be very hard to keep track of what is happening with them many months later, especially since it’s all volunteer labor. Editors please show us the courtesy of telling us what exactly you’re asking us to do!
(2) In R&R submissions, require the author to explain the changes to the article. This is a regular practice, I know, but it’s not universal. It makes reading the R&R much easier, rather than having to hunt down your previous comments somewhere on the janky journal website, or in one’s own records (if one is very organized).