Is Peer Review in Philosophy “Broken Beyond Reasonable Repair”?
Over at The Philosopher’s Cocoon, Helen de Cruz (SLU) laments her experiences with peer review from the perspective of an editor trying to get submissions refereed, saying “it is my strong suspicion that the peer review system is finally broken beyond reasonable repair.”
[Brett Weston, “Broken Window”]
It has come to the point that the most suitable referee(s) for a paper are almost never available. It takes us so long to find reviewers, sometimes a month, six weeks or more (this situation is also exacerbated by the fact that many people don’t respond to referee requests at all). This lengthens the span of time even more for the total review process. All sorts of small fixes don’t work anymore, e.g., shortening the time that people get to review, asking for alternatives (this is still very much appreciated, but unfortunately alternative referees are just as unavailable).
She is seeking alternatives to the peer-review system, not just “tinkering” (like perhaps this?) but something more radical (perhaps something like this). I urge readers to head over there to propose some.
One of the first commenters there referenced “Slow Philosophy,” a proposal Jennifer Whiting (Pittsburgh) floated here in 2015, which would alleviate a variety of problems. If you’re unfamiliar with it, give it a read.
de Cruz notes “Of course, it could be that my editorial experience is atypical—I’m just one scholar.” It might be useful to hear from others in editorial positions at journals about the extent to which their experiences match up with hers.
There is a simple solution to this problem, if only editors would take it on board. The solution is to name and shame prospective referees who refuse to review, or who do not respond to requests. This would fix the problem more or less immediately. It damages one’s professional reputation to appear like a ‘free-loader’ — one who publishes in journals but doesn’t pay their dues in serving as a referee.
[Editor’s note: Arnold changed his mind about this proposal, as he explains in a comment below.]
The logistics of this wouldn’t be very difficult: journals could simply publish to the Daily Nous each week or so a list of people who have been declining beyond a reasonable threshold (all of this can be made clear).
Legally speaking, there is no privacy violation here. Consider that newspapers can unproblematically report when someone says ‘no comment’. The person who says ‘no comment’ *or* refuses to return a call can have it accurately reported of them *that* they did not return the call. Accurately reporting that someone did not respond to a request to review is the same kind of principle.
The journal can cover its tracks a bit further by registering up front that it will be embracing a ‘sunshine policy’ of keeping a public record of who does not respond to review requests. This policy can also be communicated in the main body of each referee request.
The above might sound radical. However, we are looking for solutions here, and the pros I think significantly outweigh the cons. The only con I can think of is that some people will have legitimate reasons for declining referee requests and won’t want to be misconstrued as being lazy. This objection can be controlled for: the journal can very easily, and indeed should, publish the reviewer’s claimed reason for refusing along with the data about who has refused. Make this all out in the open! If someone has a legitimate reason (note: it might be this person’s name isn’t even published at all, because the proposal suggests a threshold policy – only serial offenders names get outed) then this legitimate reason accompanies the name. Further: if someone is a serial offender and usually gives a lazy reason “too busy” then everyone will know that they continue to give the same excuse over and over — and over time, this will rightfully make the person look lazy. This provides a deterrent for people to continuously decline with lazy excuses.
Do you think that people who are only refereeing because they will be publicly shamed otherwise are going to perform competent refereeing work? Or are they just going to take a cursory glance at every submission that ends up in their inbox and grumpily default to reject for whatever bullshit reason they can quickly muster?
The answer to your first question is: competent enough! Often, all it really takes is a quick read from someone with the relevant expertise to be able to tell whether a paper should be rejected.What we need is for those people with the right expertise to actually be willing to give the paper that cursory look over. Threats of outing would get this needed result. Regarding the second question: that is largely what people do anyway. Most referee reports nowadays are cursory and lazy. That is part of the problem that prompted exploring a solution. A referee report done by the right kind of expert but out of a motivation not to be outed is likely going to at least as competent (indeed, even if it is somewhat half assed) as what you’d get otherwise: the 14th alternative invite who is a PHD student reviewing the paper and also probably doing a somewhat incompetent job.
I don’t get it. How will a journal editor know whether or not the reason for the refusal is that they really are just overwhelmed with other refereeing work that they *have* been accepting?
I agree though that people who don’t reply at all to referee requests are awful.
prospective reviewers would be welcome to include whatever statement they want published with their refusal, including no statement. the idea is that the fact of the refusal and the reason are then made public *if* the individual passes some threshold for becoming a serial refuser. Perhaps a reasonable threshold for being outed is three refusals in a row from the same journal.
A lot of people don’t reply to referee requests because they end up in a spam folder. Probably people should be checking their spam folders every so often, but this is becoming another big problem.
This seems quite punitive and invasive. Sometimes “too busy” may be shorthand for a variety of personal or professional circumstances that one might prefer not to reveal but that don’t at all justify public shaming. This seems especially the case for those whose lives have been upended in the pandemic, of which there are still many of us. Why create a situation where invited referees are now obliged to explain personal/professional details in order to avoid public shaming? Professional life is already damnably hard enough…
Yes, it is punitive. That is the point. If someone has a good reason for rejecting the invitation to review, they are welcome on the above plan to list that, and that reason would then accompany the report of their refusal. If people stressed by the pandemic want to avoid being outed as a lazy referee, then they are welcome to do the referee work, to help make the lives better of other people who are submitted papers and *also* stressed by the pandemic.
I did understand that. Was just hoping to lodge reservations about making invited referees reportable to editors as if employed by them, obliged to detail their private circumstances to avoid shaming, while that now employer-like editor can decide to publicly post those private details, no less. Hang privacy, we need referees? You’re “invited” and now owe us an account of your private and professional life we find convincing or be shamed?
To be clear, I wasn’t referring to mere “stress” from the pandemic. If that’s all that’s changed in your life lately, I envy you.
This seems like an unbelievably bad idea. It’s unjust toward referees who are overwhelmed by already refereeing a lot and have to turn some down (some people already referee 30-40 papers a year; and they have to turn some down nevertheless); or those who are working extra jobs to supplement their poor pay; or who are caring for family or aging parents, or are themselves dealing with health problems; or who have heavy teaching loads but who are nevertheless active enough in their scholarship to be called on to referee; or who are on leave; or who have retired; etc. This is just the sort of thing that will shame people who already have too much on their plate, but frames them instead as being lazy for needing to decline. We all inevitably have to turn down some requests, unless we’re rarely asked and have lots of time (very few of us, I’d think). So the only way it’d become (sorta) fair is if a referee can demand that they not be asked in the first place; or that they only be asked say once a year… but that will make the editors’ jobs even harder. But also: your solution for a journal and its editors who are already swamped by the task of finding referees is to ask them to do even more, by tracking and publishing the names and reasons etc. of referees who decline (on what, a new website?)? So it’s both impractical and unjust?
I’m gonna guess that Johnny or Janey Tenuretrack won’t care at all about being outed as a lazy referee since their deans and tenure committees won’t care but the thought of such public shaming terrifies grad students and adjuncts. But hey let’s add another possible source of online humiliation and stress to their lives. The lazy sods have it way too easy right?
Also I take pretty much every referee request I get but if any editor had the gall to threaten me like that I’d tell him to get bent on general principles and never accept another request from journal until they got a new editor. People don’t always respond to petty coercion the way you might hope Mr. Stephenson.
If you’re right that some people referee that often (30-40 papers per year), perhaps part of the problem is that editors have too small a pool of referees. If more philosophers were asked to referee, we could spread the burden more evenly, decreasing wait times.
As someone who turned down a couple of refereeing requests in quick succession while my mother was dying of cancer earlier this year, I find this proposal absolutely horrifying.
I think the idea is profoundly misguided. Any journal that adopted it would be a journal for which I would publically state I would never referee for.
These sage words ring true yet again: https://youtu.be/5hfYJsQAhl0
I can’t stop watching this clip.
I’m an editor. I would never do this. I’m not in a position to pass judgement in this way. And yes, sometimes it takes a long time to find referees.
Same here. I am really grateful to those who referee our papers and value them highly. No intention whatsoever to start twisting people’s arms. I don’t think it would be productive, in fact I think it would be terribly counterproductive, but even if it were I wouldn’t do it.
Compensation. The most obvious fix of all.
What what money? Springer’s? don’t think so
We get compensation for publishing papers. Not from the journals, but from our employers, who use these papers as the criteria by which we are hired and promoted. If refereeing had some comparable weight (so that, say, four or five referee reports counted as much as a paper) then that would count as compensation.
Of course, making the formal measure count refereeing this way doesn’t mean that the informal reputational status would work like this, and usually the informal reputational status is much more important than the formal measures in hiring and promotion. But the formal things do count to some extent, and so we should absolutely formalize the things that don’t get counted informally.
You’re right about all of this of course. The thing is, though, that implementing the kind of thing you are suggesting would require big structural changes and lots of coordination – maybe it’s something we could hope for in 10 years if the APA set up a big exploratory committee and invested tons of time and effort. What I suggested could happen next week to fix the problem with very minimal admin. Since negligent and lazy referees is a problem that keeps coming up over and over again – usually met with ideas for big structural solutions that no one ever knows how to begin implementing — I think the time is right instead to go for a quick and ready solution that doesn’t require a bunch of structural change. (We can pursue the structural change options also of course… but best not to put all eggs in such baskets).
I don’t get how your proposal will distinguish people who have genuinely good reasons for turning down requests from those who are lazy. As Mark Wilson said, I can see “outing” folks who never reply to emails. But who is going to start checking excuses, etc.? Or maybe your view is that there are no good reasons to turn down three requests in a row from a journal (or whatever). But I think you’ll find most people find this preposterous. Good luck with this!
Isn’t this like saying we get compensated to eat and sleep… book publishers pay for reports. Why shouldn’t journals? Open access journals can be off the hook.
The comments entered thus far assume that that people don’t have good reason for declining to referee. One good reason is that they already have too many similar tasks awaiting their attention — reading job applications and promotion dossiers, writing letters of recommendation, and so on. Another good reason is that they trying to satisfy ridiculously inflated expectations for their own tenure decisions — expectations inflated by the ridiculous pace at which their colleagues in the profession publish. Have you looked recently at the CV of an assistant professor? Jennifer Whitings suggestion in “Slow Philosophy,” was spot-on.
Exactly. Thank you for saying this.
I think Jennifer Whiting’s proposal was excellent, but I wonder how it would play out within the constraints of tenure reviews as conducted by university (rather than departmental) committees. In many disciplines, there is a high volume of publishing and philosophers would need to make the case, in every such place, for this to work well. But I assume it could be done – if APA would actually do something useful for once and produce something like guide to reasonable tenure expectations for a variety of settings that people could refer to. I believe the culture of publishing tons of things that has been overtaking philosophy, already at the grad level, is really doing a lot of harm to the discipline.
Interesting idea — suggest it to the Committee on the Status and Future of the Profession!
If you think so, please feel free to do so. I am not a member of the APA.
The publishing culture is extremely harmful—to the profession and individual persons. Recently transitioned into dissertation work and rather than exploring, and diving, into a host of topics that I genuinely care about, I feel caught in a rat race of churning out publishable work, and the pressure is unbelievable. And I’m at a PGR top 9…
I do not envy you. Here is the funny thing – I am not even sure the publishing matters all that much for grad students when it comes to hiring. In my department, none of the people we hired straight (or very recently) from grad school had more than one publication and we didn’t really care about them or even took them much into account (as opposed to dissertation). It is only for mid to advanced ass profs were this was looked upon carefully. I don’t think this is atypical. so in some way this pressure on grad students seems to me somewhat bizarre. And yet it obviously exists and people have to deal with it
It seems to me that (for grad students), it started with the collapse of the job market in around 2009 or so, and the simultaneous rise in grad student numbers (that were being admitted at ever higher counts) and the need to compete more and against people longer out of grad school. The rise of post-docs in philosophy is feeding into this too (even though they are often teaching post-docs). Anyways, not sure what to do about it all.
Acceptance rates at good journals are typically lower than 10%. It can take 1-3 months to get a desk rejection, and 6+ months to get a reviewer rejection. This means that for a graduate student to get a single publication by the time they are on the job market, they either have to get remarkably lucky, write a publishable paper very early in their career, or write many publishable papers and submit them all simultaneously, such that one of them ends up getting accepted.
This means that what looks to you like very little grad student pressure to publish (look, you only need 1 publication, and we don’t even care about it too much!) actually translates to quite a bit pressure on graduate students (unless they just bank on getting lucky) and quite a few submissions to journals.
As a member of the Committee on the Status and Future of the Profession, I’ve started the ball rolling for the Committee to take up this question and seriously consider some of the complex issues involved in trying to change things for the better, whether moving to something just like “slow philosophy,” or something somewhat like it.
It seems that there are at least three distinct but related problems:
(1) The journal system is overburdened–requiring too much time to get a response, requiring too much work by way of refereeing, and for too little benefit.
(2) Graduate students and assistant professors, in particular, face far too much pressure to publish not just an article a year or an article every other year or something in that vicinity, but much more than that. It is now common to see people publish as much in graduate school as would have been required for tenure at leading institutions 30 years ago. And it is now common that people coming up for tenure have published as much as would have been required for promotion to full professor 30 years ago.
(3) This dramatic increase in publication volume results in work that is published but which is not as extensively engaged with, cited, or read; and it dramatically increases the volume of material required for evaluation and assessment on the part of letter writers and other evaluators at various stages of a person’s career (applying for jobs, evaluation for tenure, outside letters for tenure, etc.).
It may also be that these pressures result in work that is less refined, ambitious, creative, etc., but I don’t feel well placed to make that kind of sweeping judgment. I do feel that PhD students and assistant professors feel pressure to identify and publish the ‘smallest publishable units’ they can generate, and to get things out the door as quickly as possible.
There are obviously different kinds of solutions that might be explored for each of these problems. I expect that we will consider both the journal/publishing side (the supply side) and the job/tenure/promotion side (the demand side).
In both cases, there are complex collective action problems, as well as difficult informational problems. This kind of things seems well considered by something central and large like the APA. But we will need input from interested parties along the way. If you are interested in these issues (whether for or against any substantial changes), and would like to be looped in to these conversations, feel free to email me.
We will, I expect, also have more official events and discussions down the road, too. And more public invitations to participate beyond just this comment on a blog post!
To judge the prospects of institutional change aimed at slow philosophy, it would perhaps be helpful to have a study showing whether UK-based philosophers publish, on average, significantly less than their US peers. As far as I know, the REF (Research Excellence Framework) in the UK – meant to assess academics – allows people to submit only 4 articles over a period of a few years. Has this helped with perceived pressure to publish? Or is it driven by things more complex than institutional incentives?
If we’re talking about recommendations for tenure and promotion, they could include more substantive requirements to review, as part of service to the profession. It would encourage more junior faculty to seek out reviewing gigs, as opposed to waiting to be tapped on the shoulder.
Obviously this isn’t fully within the control of candidates, so the recommendations would have to be not especially onerous. But it would certainly increase the pool.
I am also curious to hear what people think about the expected compentencies for reviewers. Many people (me included) have to find narrower and relatively ignored topics to work on in order to get more publications. I often had to decline to review a paper because I felt that I was not competent enough to evaluate the quality of a paper, even if the paper is in my AOS.
I expect that if you ask the editors, they would still be glad for your review, assuming it’s roughly in your area. I get asked to referee a fair number of papers that are _in some way_ about immigration (one of the main areas I work on) but are mostly, say, bio ethics or environmental ethics papers, or something like that. (Or, I’ve refereed a lot of papers in criminal law theory, even though it’s more of a side interest of mine than a real focus.) In these cases I’ve explained the situation to the editors, and they have almost uniformly said that they are still happy for the review. Most likely this reflects the difficulty in finding referees more than an extremely high regard of my general competence, but in these cases I have gone on to do the reviews. So, my advice would be, if you have some doubts (but are not certain that you can’t do a competent job), you should ask the editor, and he or she will likely still want you to review. If not, they will be forewarned about asking for similar papers in the future.
Is there some logistical problem that would prevent journals from adopting a credit system under which you’re required to complete two referee reports before submitting a paper? I’ve seen this idea floated before, but I’ve never seen a good argument against it. People who submit the most would have to referee the most (which seems fair to me), and this would both slow down the submissions process, alleviating the overall workload for editors and referees, and increase the supply of referees.
A few benefits:
-This would bring graduate students into the system sooner. Having graduate students referee seems a better solution than forbidding graduate students from publishing, which others have suggested.
-If the system worked well enough, you could get your credits ‘refunded’ if your paper gets desk rejected (and partially refunded if your paper gets rejected after one referee report).
-Journals could keep a shared database so you don’t have to sit around for a year waiting for Journal X to ask you to referee two papers if you’ve already refereed twice for Journal Y.
Many of the suggestions that I see for how to fix the burden on the system involve incorporating referee work to the tenure process. This would do so indirectly: if you don’t referee, you can’t publish.
If I’m understanding this correctly I think it would just end up screwing over folks who don’t receive many referee requests either due to prestige or not working in a particularly trendy areas.
I don’t think prestige would be too much of an issue. People at lesser-known departments can (and would have to) contact editors to let them know that they’re interested in refereeing. It’s not as though people at prestigious departments would or could eat up all the referee work, even if a system like this were implemented. I don’t want to suggest that there wouldn’t still be prestige bias, but given everything I’ve heard about how hard it is to find referees, I can’t imagine they’d have to wait long to find opportunities.
I admit that people working in less trendy areas would be at a disadvantage, but there are a few possible solutions. The first is that people can run a deficit: if you submit a paper, you owe two referee reports, and after, say, three submissions without you submitting any referee reports, you can no longer submit new papers. Second, a weakened version of this system could be implemented wherein someone could indicate that they’d like to submit a paper to a journal, and if there aren’t any papers submitted in their area within the next month (or two, or three, or whatever time period) then they can just submit the paper.
It’s also worth noting that people working in specialized sub fields are at a disadvantage under the current system, given that it’s harder to find referees in such areas. If this system increased the supply of referees, it could be a net benefit to people in working less trendy areas.
I guess that from the perspective of a journal editor (I am one) one argument against this is that if someone sends in a brilliant paper the journal will want to publish it, regardless whether the author has ever sent in any reports or not.
That makes sense, but if the system were set up such that one literally couldn’t submit a paper without having completed two referee reports, this wouldn’t be a problem. (Unless you mean that people *should* be able to submit papers if they have a brilliant manuscript, regardless of whether they’ve completed referee reports, in which case it’s just a matter of whether you think having brilliant papers submitted as soon as they’re completed is worth the costs of alternative systems.)
I would like to renounce my previous position, after some further thought. Although I think the spirit of it is spot on (somehow, lazy referees need punished – we don’t have the money to pay them but there is ample social currency to punish), I think Amy’s comment below hits at an important and fair worry: folks with caring responsibilities (often women as it is) are going to have to decline more often than others on balance – this would mean on my policy they’d have to air their reasons more often, and there is a risk that those reasons wouldn’t be given due weight. I don’t see any obvious way around this – and so I will take it back.
That said, I’d like to throw my support instead behind a *variation* of a proposal that has appeared in the comments sense. It’s the positive credit proposal: you have to review two (two credits) before submitting one to any given journal. In theory, this is a nice deterrent approach, and (unlike the structural change stuff) could be simply implemented, which I like. BUT, there’s a problem with that proposal (which I will suggest how we can correct it). The problem is that you could still be lazy on that proposal. Senior Professor X can fail to respond to 10 emails for reviews, and then review two papers, and have their credits in hand to submit a paper. Not good! here’s a better version of the proposal: For any given journal, there should be a policy in place that looks like this: your refusals to review are counted, and once you get to three, you are banned from submitting to that journal for a year. That will incentivise people to be mindful about declining.
Or course, the same criticism could be levied: “But doesn’t this disadvantage overly burdened individuals with extra commitments in their personal lives?” yes, I think, but in a way that is much more acceptable than the shaming proposal we began with. Not being able to submit to a journal after three declines is much less punitive than shaming, and such that the disvalue of such individuals not being able to submit a paper to that particular journal in a 12 month span might even be negligible if they didn’t intend to do so anyway. Even so, it will function plausibly psychologically as a deterrent in just the way we need to offset the lazy/negligent referee epidemic we’re facing. So: the above now is my new proposal.
What about changing the expectations for how long/detailed a referee report needs to be?
I’ve often received multi-page reports with ‘reject’ recommendations. While it’s kind that these referees have taken so much time to provide feedback, it’s not clear to me that more than a few sentences are needed in order to succinctly explain why the paper is being rejected.
Likewise, “revise and resubmit” has turned into an extensive, detailed process of feedback that feels like an extension of a *pre-submission* process of receiving detailed comments from colleagues. Once more, it’s kind of reviewers to provide this level of feedback. And I’m sure we’ve all benefitted from receiving extensive feedback from referees (I certainly have). However, blurring the line in this way between pre-review feedback and peer review leads to a massive increase in work for referees.
My overall thought here is that peer review is much more time-consuming than it needs to be. If peer review were kept more sharply separate from the task of providing extensive commentary on colleagues’ work (and engaging in detailed back-and-forth exchanges about said work), it would take a lot less time, and maybe more people would be willing to do it.
I have received plenty of reviews from referees who have adopted your position that peer review can consist of a sentence or two, and I’ve also refereed for journals that have very explicitly said things like “you don’t have to write a lot, especially if you’re recommending rejection.” I think probably I’ve received more short (and, if I’m being perhaps uncharitable, perfunctory) reports than extensive reports, doubly so if we focus just on the “reject” reports. So I’m not sure that the expectation that reports be long is one that lots of people share. I often write long-ish reports but I don’t feel as if I’m expected to.
FWIW, I need to figure out what I think and I do that better when I write it out. So I write a report regardless of what the journal requires just to make sure I’m not making a bad seat of the pants judgement. (I probably still make bad judgements, but they aren’t seat of the pants.)
I have a very hard time wrapping my head around the whole “punishment” mentality… we have been talking about how refereeing work is exploitative for the lack of compensation and lack of both career and intellectual recognition. And the idea is… since some of us have little choices but to be exploited, let’s make sure we use punishment to keep everyone exploited in the name of fairness?
Furthermore, to not allow people to submit unless they have reviewed papers for a journal are giving journals a disturbing amount of gatekeeping power to the shape discipline (e.g., what topics “deserve” attention, who has a credible voice on a topic), an issue that has already been a concern. Someone who has never published anything big nor from fancy programs probably would not be invited to review in the first place, making it harder for them to find places to submit/publish to break out the cycle. One just can’t get fixated on the “greater good” and punish everyone into it…
If we reduced the number of submissions, potential referees might not be swamped with requests and could referee in a timely manner. How to reduce submissions? 1. The quick fix: I don’t see why graduate students “need” to publish – this is a recent trend. In the old days they could distinguish themselves through their PhD theses – and this would be an important factor for hiring people. I don’t have any data, but I am assuming that a good chunk of submissions is from grad students? 2. The long term fix: change the culture. The pressure to publish constantly is anathema to good philosophy.
Journal editors seem to think that scholars are the problem when they do not answer referee invitations on the same day or do not return their report in time. But a major part of the problem are the editors and/or journals and/or publishers.
Consider for instance the top journal in moral philosophy: Ethics. After two outside reviewers have recommended acceptance for a paper, associate editors of Ethics vote and reject roughly half the papers at that point. Ethics receives hundreds of manuscripts every year, so now many papers that were deemed publishable quality by independent experts are rejected and again submitted elsewhere and so the reviewer process starts from the beginning again. I think that is just an excuse to keep the acceptance rate of Ethics below 4 % or so (but please, editors of Ethics, correct me and explain that you have a good reason for this voting).
Someone once made a joke that we should start a new journal, Ethics 2. They would publish every paper without further review that Ethics rejected at their final voting stage. I would very much welcome such a journal, in fact, if someone wants to co-found such a journal please contact me. Then we would just publish everything as long as you can prove that your paper was rejected by the editors even though reviewers wanted to see your paper published (in fact, maybe this Ethics 2 could publish papers from other journals as well in case editors of those journals rejected papers reviewers wanted to see published).
I think it is a bit more complicated than that. The editors are responsible for the quality of the published articles and, hence, for the journal, not the reviewers. The role of the reviewer is always advisory not executive. It is also the case that getting two good reviews is not necessarily a mark of high quality – although in most cases the reviewers’ consensus most likely means that the paper is good, in some cases it might not. After all, these are just two people who happened to have the time to read the paper and liked it enough to recommend it and could have erred in their judgment (for many reasons). Hence, the number of not so good papers that are published (and some good ones that get rejected). It is up to the editors to make a decision and to take into consideration both their own judgment and the views of the reviewers. This is not an easy thing to do I imagine, especially for journals that have a lot of submissions and, consequently, a lot of papers OK’ed by reviewers.
Joe is right, of course. A few more specifics, since I’m an ETHICS associate editor:
The editors do consider abandoning the extra step, from time to time, but most editors want to keep it. (I guess I’m ambivalent.)
Hi Jamie,
I wonder to what extent the Ethics editors routinely rejecting expert advice might contributes to a “referee shortage”?
The last time I refereed for Ethics, it was for a paper on a topic that few of the editors worked on. Both I and the other referee enthusiastically recommended publication — on the first submission and on a revision.
The editors rejected the paper.
The editors are of course a very distinguished group of scholars, and Ethics is of course entitled to its preferred policies.
But I could see a referee in that position reasonably wondering, (a) why did Ethics ask for my assessment? and (b) why should I think of this experience as anything other than a horrendous waste of time, never to be repeated again?
I don’t know that Ethics has a referee shortage; but if the journal does, it might make sense to revisit the policy.
jmd
Hi John,
Yeah I dunno.
I don’t think Ethics has more of a referee shortage than other journals. But I don’t have a broad comparison class. I’m in a similar role at JESP, and definitely have more trouble getting reviewers for them than for Ethics, but I don’t recruit reviewers for any other journal.
How about we pay reviewers for their labor? Obviously there are a lot of details, like how much, where the money comes from, etc, but as a rule, unpaid labor is reluctant and half-hearted, done only by people who find intrinsic reward in the work (and there aren’t too many such scholars who enjoy peer-review.)
So incentivize them with cash.
Where the money comes from seems a rather important detail, not least because it’s inseparable from how much we could afford to pay, and that makes all the difference. If we paid people $10,000 per review I am completely certain we would no longer have a shortage of reviewers. Conversely, we could probably find the money to pay people $25 per review, but I doubt it would make much difference.
Conversely, we could probably find the money to pay people $25 per review, but I doubt it would make much difference.
Not least if, as seems plausible, you had to fill out a bunch of tax forms, provide bank data, etc. I have seriously considered forgoing more than $25 from (I think) Cambridge University Press for a manuscript review because their “outside vendor” forms were such a pain to deal with.
Some journals – at least ones published by Sage, though maybe others, too – do offer a sort of small in-kind compensation – free access to Sage journals for a month, or something like that. I have found the registration process to be a bit of a pain, so I haven’t made that much use of it, but such things don’t seem unreasonable, and at least offer a token of recognition.
This makes sense, but I do wonder what the 25$ figure is based on. Is this a rough guess? Based on experience? Based on anecdotal evidence? Or simply a rhetorical device to make a point? It’s a bit hard too make out how this was intended…
And what do other people think are plausible figures for compensation per review?
It’s a rough guess based on experience and anecdotal evidence, used as a rhetorical device.
If I were to flesh it out a bit: two referees for a paper would cost $50, so (given rejection rates) it would cost mid-hundreds of dollars per article to compensate referees at that rate. A few hundred dollars per published article is the kind of level that we could more or less manage with submission or page charges.
But $25 just isn’t nearly enough to significantly motivate the majority of referees (even before allowing for Matt L’s paperwork point). That assertion is mostly just a hunch, but to back it up: it takes me on average a couple of hours to referee a paper (and that’s on the quick end compared to what others tell me). So that’s $12.50/hr, lower than the usual living wage numbers you here.
Conversely, to go significantly above $25 (say, to $80 or $100) would start costing thousands of dollars per published article. And now I have very little idea where the money could come from.
(I suppose in principle (at least for TT faculty) you could build it into people’s contracts. Reduce their salary by $10k and then get the university to pay them $1k per refereed article. Good luck getting buy-in for that, though!)
Could we give reviewers one free book? Would that be cheaper? I realize the sticker price of books is higher than $25, but the wholesale price may make it manageable?
Just a very small point. The $25 would probably all go to the banks as transaction fee for Australian reviewers.
The problem is not a lack of incentives for reviewers; the problem is a glut of competent-looking submissions — where the bar of apparent competence has been substantially lowered by the deterioration of literature into formulaic “solutions” to known “problems” using slight variations on currently “hot” ideas.
Work is work and should be compensated as a matter of basic fairness. Also, if it was, the quality would increase dramatically (including their readability and constructiveness) and editors wouldn’t have to chase the same handful of experts.
Not to bag on de Cruz’s concern, which is a real one, but… Most top journals were already, well before the pandemic etc., at the stage of it taking a month or 6 weeks to find a referee to review a ms. A few years back I had submissions to Mind or Phil Review take 8-12 months just to get me their first decision (a reject, and an R&R), where the editor’s excuse was that they had a very hard time finding a referee. So I would’ve thought that the conditions she’s describing were already par for the course. And further: the very process of desk rejection, unfair as it sometimes is, is because editors often have to just decide whether to consider it at all, knowing that it may be hard to find a suitable referee. So in a way, none of this is a new problem. It may be that it’s becoming more widespread, at far more journals. But the system was already broken in these ways.
I emphatically agree with slow philosophy (maybe because I’m slow!). But I also think we should encourage a culture of reciprocity. Someone reviewed your work. If you have the competence, accept the reviewer assignment. This applies especially to tenured and more senior philosophers, who aren’t on a tenure clock. Reciprocate what has been done for you.
I think the problem is that we need significantly more referees than published papers, and it’s difficult to know *how much* reciprocity is needed. I say yes to basically all requests, and end up refereeing maybe 30 papers per year. That’s fine, but if it was 80 papers per year it’d be too much and I’d have to put down limits. But then what would a reasonable limit be?
Once per week, save for holidays. Maybe 45 per year?
Sounds like quite a good answer actually.
On one way of doing the math, 45 sounds like an awful lot.
Suppose that on average, a decent refereeing job takes one “research day” — about the time one has for research on days when one has time for research. (on average: quick rejections and acceptances may take less, serious work on an R&R may take more).
YMMV, but I’m guessing many of us get in far less than 365 research days a year, meaning 45 reviews might take 20-25% of one’s research time.
Is that the suggestion for an appropriate refereeing load?
As you all think about this and propose numbers, what teaching load do you have in mind?
I was suggesting it for David Wallace, whom I’m guessing has less than a 2/2. I have a 4/4. For me, I would think 12 a year is good. I only submit about 1-3 articles a year anyway.
Robert’s right that I have quite a low teaching load.
That said, I usually budget about 2 hours on average for referee reports, quite a bit less than Doris’s estimate.
Thanks to the both of you. This helps.
I’m not an academic and know nothing about the refereeing process beyond what I’ve read here and in other similar places.
But it seems to me that these problems are a consequence of a more fundamental problem, which is that there are too many papers written, because academia (not just philosophy) targets/incentives a maximand of numbers of papers written and published, and then cited by other papers written and published &c &c.
This quantity theory of scholarship seems to me to be bs, but the consequence of it is a ridiculous demand for people to review the ridiculous number of papers that people are writing.
‘Zackly
There’s pretty compelling research in science studies (associated with Simonton) suggesting that quantity predicts qualtity — where quality is understood as the citation impact of individual papers.
Roughly, the theory is that impact is a substantially stochastic process, so more papers maximizes your chances of a “hit” individual paper. The “perfectionist:” strategy of producing fewer, better papers results in fewer chances to get lucky.
Not saying anyone should care about “hit” papers (though their dean might!), but if one has an interest in producing them, they may have good reason to go in for “fast philosophy” and publish more.
I’m not an editor, but I don’t doubt for a second the truth of Helen de Cruz’s claim. But what makes this especially frustrating, from the point of view of an early career scholar trying to find work enough to pay the bills, is that it seems impossible… How can there simultaneously be both a lack of available referees, and a glut of people with phds, such that there are far too few jobs to go around? Why aren’t we (by “we,” I guess I mean the journals) hiring these new phds to be full-time referees? Are they not qualified, not expert enough? Is it supposed to be “peer” review, or what?
To me a big part of the problem is the way refereeing is regarded. We see it as a thankless mindless chore like the academic equivalent of loading the dishwasher or cleaning the bathroom. It needs to be done but it’s a mindless chore that anyone could do. The truth is though it’s actually a pretty big honor to get asked to referee by a prestigious or even very solid journal. And you get considerable power to shape the discussion around an issue. Beyond all that a good referee can make a huge difference in a paper. I’ve got a paper on police interrogations coming out soon that’s massively better in its revised form because of some spot on suggestions from the referees.
I’m not sure this would magically solve the problem but I think trying to shift professional norms so that the respect and perceived importance of refereeing matches its real importance might help. To that end it seems like more journals recognizing exemplary referees in some way or other would be good. I might even suggest adding some refereeing requirements for tenure or promotion. This wouldn’t just be patting people on the head for being a good boy or girl. After all, getting asked to referee on subject X is a pretty good sign that one’s work on subject X is taken seriously. That seems a solid data point for deciding one’s scholarly impact. Not to mention that refereeing itself has such an impact if done well.
Philosophy doesn’t need journals, or any sort of official peer review. Academia does, but philosophy doesn’t. Philosophy could get along just fine with authors putting papers up freely available on the web with a proper versioning if they revise the paper (in response to comments, criticisms, etc.).
And, of course, philosophy doesn’t even necessarily need papers (or only papers). Philosophy can be done via video, audio, tweet thread, blog, etc., etc.
I responded to your previous article about journal peer review from my perspective as a philosophy book editor for a university press. I concur 1000%. I’d be open to any and all alternatives with the proviso that our faculty board would need to agree to them. Board members seem to be in love with peer reviews.
Wendy Lochner Columbia UP
My experience did not match up with Helen’s. About two-thirds of my referee requests resulted in agreements to referee, and in five years no more than four or five of those resulted in no report delivered. (In my time as editor I received about 850 referee reports.)
I don’t know how important these things are. But (1) I sent referee requests myself, rather than having them shot out of the maw of an Editorial-Central type system. It is much harder to ignore a message that looks like it came from a human. That cuts down a lot of time due to potential referees ignoring requests. (2) I drew on a very large pool of referees. Those 850 reports involved about 400 unique referees. (3) The journal I edited was one that is non-profit and now open-access, and many of the referees view it as an ongoing common project rather than just a place to get visibility for one’s work.
Also: it would help if none of us were willing to referee for a for-profit journal. I don’t know what proportion of the refereeing work is done for for-profit journals, but it should be none, so that other journals have more available referees and that the for-profit journals die a painful death.
I like this advice. Which journals are for profit and which are not? Is there a list or easy heuristic?
I believe Elsevier, Wiley-Blackwell, Springer, and Taylor & Francis are all for-profit.
Here is a list of just Springer philosophy journals. Note some of the highlights: Phil Studies, Synthese, Erkenntnis.
GSTF Journal of General Philosophy (JPhilo)
Asian Journal of Philosophy
The Journal of East Asian Philosophy
Frontiers of Philosophy in China
Review of Philosophy and Psychology
Philosophy & Technology
Philosophy, Ethics, and Humanities in Medicine
European Journal for Philosophy of Science
History and Philosophy of the Life Sciences
Philosophy of Management
Biology & Philosophy
Criminal Law and Philosophy
Journal for General Philosophy of Science
Medicine, Health Care and Philosophy
International Journal for Philosophy of Religion
Studies in Philosophy and Education
Journal of Indian Philosophy
Continental Philosophy Review
Law and Philosophy
Linguistics and Philosophy
Annalen der Philosophie
Zeitschrift für Ethik und Moralphilosophie
The Journal of Ethics
Journal of Dharma Studies
Annalen der Philosophie und philosophischen Kritik
International Journal of Dharma Studies
Human Studies
Jus Cogens
Res Publica
Medicine Studies
Sophia
Metascience
Dao
Acta Analytica
Philosophia
Topoi
Metaphysica
Husserl Studies
International Journal of Hindu Studies
Teaching Business Ethics
Philosophical Studies
Brain and Mind
Recht Innovativ
Axiomathes
Knowledge, Technology & Policy
Leviathan
Human Rights Review
Ethical Theory and Moral Practice
Science and Engineering Ethics
Acta Juridica Hungarica
Minds and Machines
Indian Journal of History of Science
Law and Critique
Journal of Indian Council of Philosophical Research
NanoEthics
Identity in the Information Society
International Journal for the Semiotics of Law
BMC Medical Ethics
Health Care Analysis
Foundations of Chemistry
Synthese
Theoretical Medicine and Bioethics
Argumentation
Phenomenology and the Cognitive Sciences
Studies in East European Thought
Foundations of Science
Erkenntnis
Food Ethics
Life Sciences, Society and Policy
Fudan Journal of the Humanities and Social Sciences
Asian Journal of Business Ethics
The Journal of Value Inquiry
Hague Journal on the Rule of Law
Research Integrity and Peer Review
Negotiation Journal
Acta Biotheoretica
Asian Bioethics Review
International Journal of Ethics Education
Poiesis & Praxis
Science & Education
Journal of Cultural Cognitive Science
Mind & Society
Humanistic Management Journal
Neuroethics
Research on Language and Computation
Biosemiotics
International Journal of Value-Based Management
Jindal Global Law Review
Theory and Society
Interchange
Theory in Biosciences
Journal of the History of Biology
Agriculture and Human Values
Journal of Logic, Language and Information
Natural Language & Linguistic Theory
Journal of Mathematics Teacher Education
Monash Bioethics Review
Artificial Intelligence and Law
Social Justice Research
Foundations of Physics
Archive for History of Exact Sciences
Studia Logica
Natural Language Semantics
Deutsche Vierteljahrsschrift für Literaturwissenschaft und Geistesgeschichte
Journal of Happiness Studies
Contemporary Political Theory
Journal of Agricultural and Environmental Ethics
Journal of Philosophical Logic
Journal of Business Ethics
I’d love to have more OA journals to submit to and fewer journals owned by giant for-profit publishers. But the journals themselves are not for-profit. They happen to be owned by these corporations. Maybe that’s bad, maybe they could opt out, I don’t know. But it seems a little harsh to suggest that these journals, which are (often well) run by our colleagues are for-profit machines. With that said, I share your sentiment. I just wish Ergo and Phil Imprint will accept my papers but just my money. (I love you guys)
One wrinkle: although AJP is published by T&F, it is not owned by them. The AAP (Australasian Association of Philosophy) owns the journal, and enters into a contract with a publisher (currently T&F). The AAP’s own income from AJP is used to support various worthwhile aspects of professional philosophy in this part of the world.
Whatever the virtues or vices of driving for-profit journals out of business, I don’t think it would help the refereeing shortage – unless we had reason to think that new not-for-profit journals wouldn’t replace them.
I agree that the primary reason for refraining from refereeing for for-profit journals is not that it will lower referee demand overall. But it sure seems very implausible to me that all of these journals, which bring in huge profits for Springer et al. by way of uncompensated referee labor, would be replaced should they be killed by refusal to referee for them. I will not name names but not all of these are Phil Studies and it would not be surprising if they simply disappeared were they not cash cows for Springer.
Fair.
If these journals were to disappear, the not-for-profit journals would not only inherit the refereeing work that is now done for them, but also all the submissions that they would have received otherwise. So there still would be a shortage of referees. I think the future must be community-owned open access journals, but that’s not going to help with this problem.
Forgive my ignorance, but what’s wrong with for-profit journals? Is it just that: (a) they employ a bunch of referees who labor for them for free while (b) they make money by selling articles? Or is the idea that academic journals should all be free? Not against any of that! I just have never heard the claim that for-profit journals shouldn’t exist.
“Also: it would help if none of us were willing to referee for a for-profit journal. I don’t know what proportion of the refereeing work is done for for-profit journals, but it should be none, so that other journals have more available referees and that the for-profit journals die a painful death.”
I don’t like for profit journals for a lot of reasons. But we have to ask whom this policy would harm. At least in the medium to near long term it’s clear that it would harm those who submit to those journals who will have to deal with an even more dysfunctional peer review system, and a disproportionate level of harm will accrue to grad students, untenured, and non-TT faculty who need to publish. Established academics who don’t need to publish or the very least have more routes to do so and can wait longer wouldn’t suffer nearly as much. And I know you might counter that people shouldn’t submit to journals run by for profit companies but in the real world things aren’t that easy. There are more than a few journals that are put out by for profit companies that have very good peer review processes. The Southern Journal of Philosophy comes to mind. I’ve never been kept waiting more than two months there and have always gotten useful feedback. I wouldn’t in a million years say that about some other journals. This whole suggestion just seems typical of a lot of things academics do in the name of social justice. It doesn’t require them to give up anything– in fact it makes their lives easier since they can shirk doing work– and actively harms folks less fortunate than them.
This is why I left the field – in the end we’re all just interested in our own ideas, and the review process is yet another reflection of this fact, that reading peer submissions is viewed in the light of “What can you do for me”? There’s been an explosion of papers and yet a lack of progress because, in philosophy, faced with bills and children, wisdom is no longer loved.
For what it’s worth, I really enjoy reviewing, and I enjoy it for the same reason I enjoy reading philosophy. It’s exciting to become acquainted with what other people think about topics I care a lot about.
I doubt I’m alone, and I think the explosion of papers is that the job market is really tough and people need to publish to get hired, and since you also need to publish for tenure, this ups the amount of papers people on the tenure track have to try to publish, etc. I don’t think somehow philosophy only attracts people who are just interested in their own ideas.
Remember, the people applying to graduate school are still by and large undergraduates who have become fascinated with the field largely through reading stuff written by others.
I think this is a good attitude to have. But it is hard to keep as one gets more and more requests and more and more other bits of work to do. Still, I think this is a good attitude to have and that is the main thing I want to say here.
Can I just add that the problem is exacerbated somewhat by journal editors who should be doing more desk rejects, but instead pass stuff along to referees. We’re all busy telling our graduate students that they need to publish before graduation, but it means journals are getting piles of submissions that are obviously lightly-revised papers written for graduate seminars. I’ve been asked to referee enough of these that I have begun to push back a bit against editors who seem to be asking me to do their job for them. I was recently told by an editor “I just wanted to make sure there wasn’t something in there that I was missing,” and I was like, “you shoulda risked it…”
I am a tenure track professor and I once got a referee comment on one of my papers (researched, conceived, and written entirely here at my university) saying that my paper was clearly a lightly-revised section from a dissertation or something like this. My paper was from an different subfield than my dissertation work and had nothing to do with anything I did in graduate school.
This is not to impugn your abilities to recognize whether something is or isn’t a lightly-revised paper written for a graduate seminar, since maybe you actually can sniff this out with more acuity than the referee who wrote the report in question. But not everyone who thinks they’re able to judge this thing is in fact able to do so.
I remember from some discussion of peer review at some point somewhere (online, I think, maybe here or The Philosophers’ Cocoon) someone made a similar point about how people often think they can guess who wrote the paper they are reviewing, but they are often wrong. I think some referees imagine they’re better at discerning these things than they in fact are, especially since you often get no chance to confirm your suspicions.
Over at Philosopher’s Cocoon someone made the (excellent, I thought) suggestion that editors use the following to find well-qualified reviewers outside their own network:
https://philpeople.org/find-philosopher
I thought it deserved sharing here as well. Some more details over there.
I submit papers to psychology journals more often than philosophy journals, and these issues do not seem to be nearly as bad in psychology.
The occasional irresponsible or incompetent referee report happens in psychology, as does the occasional frustrating delay, but in general nothing nearly so bad as what seems to be typical in philosophy. I don’t follow forums like this in psychology, but the psychologists I talk to do not seem nearly as upset about the journal system as do many philosophers.
I don’t have a hypothesis about the difference — though the high rate at which many psychologists publish leaves me a bit skeptical that the problem for our discipline is “fast philosophy”
Nor do I have opinions about whether the journal system is in need of a major overhaul. But the fact that psychology seems to do considerably better with what seems to be a pretty familiar system has me wondering about the need for such an overhaul in philosophy.
I’ve been told that in various sciences people roughly know where to send a paper to have it accepted. I suspect that reflects more consensus in the field about what is good work and what is important, and about how good and how important. But philosophy is the stuff about which there is less agreement about methodology and importance and so on. (We are doing what did not spin off as a special science yet.) So it does not surprise me that papers bet more uptake where they are sent and bounce around less. But I would not really want to cull the ones that bounce a few times (every one of mine). I think it goes with the field.
David V is right that the problem is too many submissions but every solution to that problem at the level of the discipline creates its own problems with equal opportunity and access to the field. The slow philosophy idea is a good one, but we can’t adopt it without buy in from administrators at higher levels who mostly don’t understand philosophy anyway. And P&T committees where many people think counting citations and publications is what matters. So we are somewhat screwed.
While I’m ranting, the philosophy does not need journals refrain is unrealistic. We have morphed from a patronage/church/rich guy system for doing philosophy to one in which academia and publicly funded academia supports our work. (That’s good given the broadening that brings with it.) We have to find our way in that system which so far includes tenure and promotion (though to its detriment not as much as it used to) and that means some way of vetting our work in a way that outsiders will recognize. So I really wish there were more solutions on the table that fit with this point in the history of philosophy.
The sciences generally also have more journal specialization, something I frequently suggest philosophy could usefully follow.
Do journals make any money? Do they have any to spare? I have no idea. Suppose they do. Small amounts of compensation may not be enough to entice more reviews, but could journals afford to keep a few dedicated reviewers around? Employ a few people, at least part time?
Also, do most disciplines expect reviewers to provide comments? Most literary magazines, for examples, reject most submissions out of a slush pile with nothing but a form letter. Put the burden on authors to find critiques before they submit to journals. Reviewers should serve the journal rather than serve the submitting authors. That could speed things up.
How about publishing rejections? That is, introduce a general policy, that if a submitted paper is rejected post-reviews, the journal publishes reviewer comments, the editor’s decision and perhaps the submission on the journal website.
Combine that with a widespread policy of not accepting for review papers that have been a) rejected X number of times and/or b) for good reason and/or c) in the recent past. The result would be a dramatic decrease in submissions, an increase in the quality of submitted papers (since the costs of rejection have gone up), and an increase in the quality of reviews (for both of the former reasons, as well as the publicity of the reviews).
Of course, this would require some difficult coordination, and face resistance from those accustomed to or adept at exploiting the existing system. However, it would directly target the root of the problem: that the current system encourages the repeat submission of low quality papers (I say this as a reviewer for a fairly large number of journals), in the hopes of eventually getting lucky with one journal or another.
I think refereeing is a bit too noisy to make that a good idea. Plenty of good papers get rejected for bad reasons. If that happens to your paper, you can just shrug and send it to a different journal – but it’s another matter if it’s public.
Conversely, if I thought my referee report was a matter of public record, I’d probably want to spend vastly more time writing it. Which maybe is good, other things being equal, but I’d do a lot less refereeing if each individual report took much more time.
I agree. I have heard (and happy to be corrected if this anecdote is not accurate) that Jason Stanley’s book “Knowledge and Practical Interests” was based on a paper that was rejected 11 times. Even people who disagree with subject-sensitive invariantism about knowledge attributions agree that his book is excellent. If *that* kind of work can get rejected that many times, we should be suspicious of putting caps on rejection numbers.
I keep reading about that urban legend. I don’t doubt that Stanley’s article was, in some form, rejected that many times (I recalled 9, FWIW, but imprecision is the stuff of legends). And his book is excellent. But his book was not rejected 11 times, nor was the best version of the article. The referees did not reject the article that we know; they rejected worse versions of it. Manuscripts change drastically through the submission-rejection-revision cycle. I’ve had papers rejected many times before being published but it’s never occurred to me that the first iteration deserved to be accepted as much as the accepted iteration did. (I recently retired a paper after 12 rejections—do I think it deserves to be published? Yes. Will I keep bothering (potentially the same) overworked referees with it? Not for a little while.)
This is not to say there should be a cap—it’s probably a bad idea; refereeing is very noisy. But I’m skeptical of the ‘some works of genius get rejected twenty times before getting their due’ myth. In fact, I suspect it may encourage stubborn submission of subpar work by people who think highly of themselves and fail to see why they should be deterred or why they should bother to alter their work of genius.
[See edited version below] I’ve sometimes refereed 30+ papers in a year. But I can no longer do that now. My employer expects me to teach 8 hours per week. I have a book under contract. I am leading a 400K project that funds a fantastic post-doc (not on the topic of the book – the book will be the main output of the previous 400K project). I’m co-editing two special issues (also linked to the previous project). I have a PhD student and I supervise some BA and MA theses. I do my share of admin work. I’m applying as co-PI for more projects, to support early career scholars. I have two small children. I’ve organised and co-organised many conferences. There’s no way all of this can fit in 40 weekly hours. Or even in a week. So refereeing papers is one of the first things that goes. I still review 5 or 6 papers per year. But it’s not manageable for me to do more. The main problem here is my employer, who thinks I can teach 8 hours per week while being the principal investigator of various research projects. But, I think, they’re wrong. My workload would be out of kilt even without the children. With the children, it’s insane. So yes, I very much welcome the slow academia proposal. And I don’t think books or threats would help me write more referee reports. What I would need is an amount of teaching that’s commensurate to my research commitments. (8h per week is double the weekly average teaching load of my Italian colleagues. 6h is probably closer to the international standard. One should arguably get 2h off when awarded mid-sized projects such as mine.)
[Note from JW: Julien Murzi asked that the following edited version of the foregoing be included here.]
I’ve sometimes refereed 30+ papers in a year. But I can no longer do that now. My full load before project discount is 8 academic hours per week. (It’s 6 academic hours after discount, but given the way things are set up here this amounts to up to 3 preparations.) I have a book under contract. I am leading a 400K project that funds a fantastic post-doc (not on the topic of the book – the book will be the main output of the previous 400K project). I’m co-editing two special issues (also linked to the previous project). I have a PhD student and I supervise some BA and MA theses. I do my share of admin work. I’m applying as co-PI for more projects, to support early career scholars. I have two small children. I’ve organised and co-organised many conferences. There’s no way all of this can fit in 40 weekly hours. Or even in a week. So refereeing papers is one of the first things that goes. I still review 5 or 6 papers per year. But it’s not manageable for me to do more. The main problem here is my employer, who thinks we can be research active and teach 8 hours per week. I think they’re wrong. My workload would be out of kilt even without the children. With the children, it can be hard to cope. So yes, I very much welcome the slow academia proposal. And I don’t think books or threats would help me write more referee reports. What would help is a full teaching load that’s more in line with the international standard. (8h per week is double the weekly average teaching load of my Italian colleagues. 6h is probably closer to the international standard. One should arguably get 2h off when awarded mid-sized projects such as mine. Classes in the German-speaking world typically meet 2h per week, so typically n hours = n/2 preparations.)
Seriously, everyone imagines themselves to have more going on than everyone around them. It’s just part of how we are epistemically situated—I only think you are doing what I see you doing. This is a major part of the problem, I think—I imagine myself to be busier than everyone else, but I don’t see their ailing parents or spouse, their disabled children, the independent studies they are doing, the hours that they spend on various projects, grant-funded or not, the reading groups they run, and so on. Part of being a productive member of a community is not supposing everyone around you to have more time than you do.
I don’t think I’m busier than my colleagues, and I don’t think I said that. We’re all too busy. That’s precisely the problem.
We’re not too busy to submit papers to journals, apparently.
There are plenty of things we’re not too busy for, including doing the research that I was hired to do, the teaching I was hired to do, the various ways in which I have to service the discipline as I was hired to do, etc. But I am too busy to do it all, and I agree with Julien Murzi that I am simply too busy to review more than half-a-dozen papers every year. I doubt I am busier than anyone else. The best way our discipline could be served is by taking, e.g., Jennifer Whiting’s suggestion, not by urging people to stop “supposing everyone around you to have more time than you do.”
Well, PhD students *must* submit papers if they want to land a postdoc; postdoctoral researchers *must* submit papers if they want to land a job; TT faculty *must* submit papers if they want to get tenure; tenured faculty *must* submit papers if they want to get promoted etc.
Not implicating anyone here—But imagine someone saying “I’m too busy” when that person is submitting and publishing, but not refereeing at least approximately 2-3x as often as s/he submits papers—that’s pretty lame. Because then “I’m too busy” just means “I’m not going to do my fair share, but instead prioritize the things that will advance my career, because I’m (somehow) a special case”.
I agree that simply blaming such a person for being a selfish jerk doesn’t change the incentive structure, and so is (somewhat) irrelevant. But still—some people out there appear to be making an exception of themselves, since people are prioritizing research and padding their own CVs over the service work generated by that kind of thing.
Ops – clicked ‘report’ by mistake. I don’t seem to be able to unclick it.
I see what you mean, of course. There may well be colleagues like this. I’m not one and most of the colleagues I know aren’t either. But I live in a small world.
In short, I think there will always be people who behave badly. And I’m not sure introducing punitive measures will make them behave in a better way. Rather, it will likely annoy the majority of colleagues, who don’t behave badly.
I think concrete measures aimed at slowing down things, such as the ones suggested by Jennifer Whiting, would help.
In general, I’m not a great fun of punishments. Perhaps I should be. As a guest editor of 5 special issues now, I know what it means when a referee agrees to review a paper and then disappears, leaving you in the cold 6 or 8 weeks later, with an author waiting.
Sometimes people are really too stressed or burned out. And the occasional slip can always happen.
But, lately, three or four referees simply went awol.
So some referees go awol; some authors just write papers and don’t waste time refereeing other people’s papers.
I guess my thought here is that if one lacks basic decency at this stage (we’re all adults), there’s little we can do to change them.
I also assume that the majority of our colleagues is actually pretty decent. So I’d rather focus on the issues faced by the majority of colleagues who, in my experience, tend to be pretty stressed and overworked.
Before tenure, I would teach 4h per week. Now it’s supposed to be 8h. I live an OK life, but before tenure it was actually pretty good. The amount of time I now have for research, refereeing, students, family, everything, has plummeted.
Here’s a proposal I haven’t seen discussed much: abandon the r&r.
Referees give a straight yes or no verdict, maybe with a multiple choice option for ‘no’ (referees could check a box to indicate whether the issue was the topic, the writing, the arguments…). If there are too many ‘yes’ verdicts, use a lottery system (perhaps organized by subfield) to decide which ones get published.
This would obviously make refereeing a lot less time consuming. And authors can seek out other sources of feedback. Is this a perfect, or even great, proposal? Nope. But if peer review is truly broken, maybe it’s better than nothing?
From my corner of the world this looks like a mistake. My favorite of my own papers got lots of help from able and generous referees.
As an editor, my experience is that the referee’s reports that are most useful are those that include real critical information and advice. (Imagine getting a split set of reports with no content and figuring out what to do.) Why shouldn’t the author get that same feedback? And why should they not get a chance to respond by rewriting? The papers I’m happiest with accepting are the ones that get better in this process and turn out really excellent as a result.
The common refrain that the barrier to slow philosophy is higher admin, or tenure and promotion committees, is overblown. Admin and tenure and promotion committee do not take univocal views about what constitutes quality and productivity in research. They tenure dancers, choir conductors, composers, and poets as much as they do economists and biochemists. If the top R1 departments collectively decide they will review, solicit letters for, and hire/promote on the basis of the best 5 papers or 175 pages of a candidate (or whatever), then university admin will accept, and all of the prestige-economic incentives in the rest of the field will follow suit. The barrier to slow philosophy is likely more a cultural problem in the field, a much harder coordination problem to solve.
And even if you don’t think it’s likely that Deans will apply the same standards to philosophers as dancers, Barry’s point here is entirely right. Plenty of humanities folks get tenure primarily on the basis of a book, and when you look inside the book, it is often 2-3 (long) chapters of original work with a survey-ish introduction and a summarising conclusion. There isn’t much more in a lot of humanities tenure files than the folks pushing slow philosophy want – it’s just in book form not paper form. The idea that Deans wouldn’t let philosophy have the same quantitative standards as the other humanities seems bizarre to me.
I love the slow philosophy idea, but: my own university admin and P&T committees often overrule departments and external reviewers. Really, what they want to see is a dossier with lots and lots of fancy publications. Everywhere is peculiar, and I’ve no doubt that other departments might have more luck. But it wouldn’t work here, and I reckon we aren’t alone.
I’m with Weary in being less sanguine about administrations simply accepting whatever standards for personnel actions departments care to set (eg, “philosophy is not a book discipline”).
Administrations can and do exercise independent judgment, as those of us who have seen administrations reject departmental recommendations for promotion and appointment can attest.
No strong opinions about “slow philosophy” here, but I’d not want to underestimate the difficulty of selling the idea to administrations, if the CVs that result from practicing slow philosophy diverge too much from a generalist’s sense of what a credible CV for the position in question looks like.
Of all of the suggestions here, reducing the expected length of the reviews is the easiest way to get more reviewers. Why does a reviewer need to write 3-4 single spaced pages to justify a “reject”? 2-3 sentences should be enough. If the verdict is an r&r, then a longer response would be justified to explain exactly what the reviewer thinks need to be revised.
As best as I can tell, too many authors use and otherwise game the review process to get feedback on their papers. Send a somewhat finished paper to Journal 1, get substantive comments back, revise in light of those comments, send revised paper to Journal 2, etc., until it gets published. Helping people to revise unfinished papers should not be the purpose of the journal review process.
I took a half hour to collect a bit of info from Ergo that I thought would be useful.
For the last 39 papers that (i) were submitted more than a week ago and (ii) were not desk-rejected or resubmitted R&Rs, we have had 66 accepted requests and 96 declined requests. That is fewer than three declined requests per paper, and about a 40% accept rate. (I excluded R&Rs because the rate of acceptance of those requests is much higher, probably 90%.) Six papers have had five or more declines. No paper has had more than six declines. (I do recall one paper last year having around 20 declines. When that happens it can certainly feel like things are broken. My sense is this is very rare though.)
To me, this does not seem like a crisis requiring anything at all to be done differently, let alone things as drastic as some people are suggesting. But of course other journals might be having different experiences. Maybe some people are accepting requests from Ergo because we are open-access. If that is the case, then an obvious solution for other journals presents itself.
Another contributing factor may be that reviewers know that Ergo desk rejects about 2/3 of all papers and that the paper’s have been read by an editor with an appropriate AOS, which means that reviewer can a higher level of quality of the papers they get to review.
One modification to one of the already suggested solution is for the worldwide philosophy society to set up a budget to hire a large professional team of reviewers, who maybe students or philosophy freelancers.
However the long terminal solution as an adaptive response to the ever exponential growth of information–not just in philosophy but in all fields–is to invest in artificial intelligence. Before scoffing at this outrageous idea, please consider recent AI significant progress where it can read and summarize technical papers and participate in debates. In the long run, this may be the only solution to pm human survival and progress.
Honestly, given the number of emails that go unresponded to in this profession, I wouldn’t be surprised if a lot of referee requests just fell through the cracks.
I think more papers need to be assessed in house before being sent to external reviewers. I understand editors are busy, and have limited expertise, which is why this might not be possible at the moment. So why not dramatically increase the number of editors/associate editors at journals? At the moment, these positions are hoarded by the select few who successfully apply or are asked to take such roles on. I have a hunch that a lot of established academics would be happy to commit time to doing this because these kind of positions, unlike reviewing, do make a difference when one applies for promotion etc. as they considered indicators of professional standing/esteem.
So long as journals commit to triple blind reviewing I don’t think this should raise particular concerns about the process.
I think you bring up a great point, James. The structure and size of the editorial team matters, and perhaps public discussion of the ‘publication crisis’ should focus on this aspect of the review process. How is the editorial process of ‘successful’ journals structured? How many EAs are involved? We know some journals are able to get comments to most authors in a timely fashion (AJP, for instance). What’s their secret? Instead of suggestions to wildly change the way peer review works (with prizes, or paid referees, etc.), let’s look at how these journals are actually managed and run.
I am reposting (with slight amendments) one of my contributions to a thread on the same topic on NewApps in 2014:
Catarina [Dutilh Novaes} suggests that if you are a fairly established member of the profession (e.g. not a graduate student, adjunct etc.), and you submit X articles in a year, you should do 3X referee reports: 2X to cover for your own submissions, and an extra X for the surplus of people not in a position to act as a referee themselves. ‘Does that sound reasonable?’ she asks. Well in one way it does and in another it doesn’t. It would be a reasonable refereeing workload for an established member of the profession, and if that were all that they were expected to do, they would probably do it promptly. The problem is that as things stand, the system would collapse if academically successful philosophers did not do a great deal more refereeing than that. And it is *because* they often get asked to do a lot more than that, that refereeing burn-out sets in, that the as-yet unrefereed papers pile up and that the reports when they appear are often snarky, superficial and dismissive.
First point to note. The top fifty journals have rejection rates varying between 80 and
95%. This means that the vast majority of PUBLISHED papers are rejected the
first, the second and even the third time around. This suggests that the average PUBLISHED paper has generated at least six referee’s reports and that there are probably quite a lot of papers that never see the light of print but
which have generated many more.
Second point to note. You get asked to referee a paper under one or more of three related conditions:
1) you have published something in the area that has garnered a few citations or has
otherwise created a stir,
2) your own work is cited in the paper to be refereed,
and
3) you have an established reputation as a reliable referee.
You don’t have to have to have published all that much in a given area to meet conditions 1) & 2). For example, I have only one publication on truthmaker theory (a
coauthored paper with my former colleague Colin Cheyne), but it is reasonably
well-cited, so I am often asked to referee papers on truthmakers and negative
facts. I have other papers in other areas that to my mind are equally good but
are relatively uncited. Hence no refereeing requests.
Obviously you don’t get to meet condition 3) unless you have already done a bit of refereeing, usually, though not necessarily, for the journal that makes the request.
Third point to note (a corollary of the second): Editors not unreasonably use citations and citation counts to help them select referees. (This is not unreasonable since
simply having published in some area does not mean that your publications will
be any good or that you will be a competent referee. Citations are at least indicative of the respect of your peers.) But this militates against the selection of younger referees,
since it often takes time to acquire citations.
Now imagine a successful philosopher, call her Sophie, all of whose papers get accepted the first or the second time round. Thus the average number of referee’s reports per published paper that SHE generates is three. Since speedy acceptance is at least roughly correlated with quality, her papers are generally good and are regularly cited in the area or areas in which she works. This means that she often meets conditions 1) & 2) above. Let’s imagine too that Sophie is a conscientious person eager, at least initially, to do her bit for the profession. So she soon acquires a reputation as a reliable referee. Pretty soon the referee requests come flooding in. She gets a lot more than the nine requests per published paper of her own that Caterina’s formula would suggest.
Now imagine aless successful philosopher, call him Jack. None of Jack’s papers are accepted the first or the second time around. The average number of referee’s reports
per published paper that HE generates is nine. Furthermore, since tardy
acceptance is at least roughly correlated with poor quality, his papers are
pretty ho-hum and are NEVER cited in the area or areas in which he works.
(Remember that four years after publication, the median number of citations per
philosophy paper is ZERO – that is most papers are not cited AT ALL.) This
means that Jack NEVER meets conditions 1) & 2) above. Because he never
meets conditions 1) & 2) , he is unlikely to acquire a reputation as
reliable referee even if he is a conscientious person with a knack for seeing
what is good or bad in other people’s papers despite deficiencies of his own
work. Is Jack really going to get the 27 refereeing requests per published
paper of his own that he would need to complete in order to meet Catarina’s
formula? Obviously not.
Thus the problem I suggest is this. For structural reasons, the burden of refereeing
falls disproportionately on the more successful members of the profession who
are the very ones who generate the lowest numbers of referees’ reports. The
things that Catarina complains of – refusing to referee and not even answering
requests – are in many cases symptoms of burn-out rather than instances of
depravity.
One more thing. It is often suggested that the problem could be partially solved if overburdened referees recommended substitutes, especially junior colleagues or graduate students. But this only makes sense if the overburdened referees are well-networked and well-connected with junior colleagues and graduate students to recommend. But what if there isn’t a very strong correlation between being well-cited and being well-connected? I am reasonably well-cited in several subject areas and get asked to do a fair bit of refereeing. But I live at the ends of the earth and work at a department with only a few graduate students who do not work in these areas. Consequently I am not very well connected. I can only recommend substitutes in *one* of my areas of expertise (namely conspiracy theories). As for the others, I haven’t a clue. Perhaps I am unusual in this, but if I am not, this might explain the frustrations of an ‘associate editor’ over on the Philosopher’s Cocoon. He/she complains that ‘when I do receive suggestions [for substitutes], about half of the time they aren’t helpful because they are completely obvious’ Perhaps , as in my case, this is because many first choice referees are not sufficiently well-connected to know of any non-obvious substitutes to recommend.