How to Fix the Referee Crisis in Professional Philosophy (guest post)
In the following guest post, Eric Schliesser (Amsterdam) provides a two-step solution to the referee crisis in philosophy.
A version of this post first appeared at Professor Schliesser’s blog, Digressions & Impressions.
[Theo Van Doesburg – Contra Construction]
How to Fix the Referee Crisis in Professional Philosophy
by Eric Schliesser
In a recent post at The Philosophers’ Cocoon, Helen De Cruz (SLU) writes:
It has come to the point that the most suitable referee(s) for a paper are almost never available. It takes us so long to find reviewers, sometimes a month, six weeks or more (this situation is also exacerbated by the fact that many people don’t respond to referee requests at all). This lengthens the span of time even more for the total review process. All sorts of small fixes don’t work anymore, e.g., shortening the time that people get to review, asking for alternatives (this is still very much appreciated, but unfortunately alternative referees are just as unavailable).
Of course, it could be that my editorial experience is atypical–I’m just one scholar.
But it is my strong suspicion that the peer review system is finally broken beyond reasonable repair. We’ve seen a slow worsening of the situation and the pandemic has finally broken the system. People are burnt out and overburdened, job candidates increasingly desperate.
De Cruz’s piece has generated a huge discussion, not just among the Cocooners, but also here at Daily Nous. The situation she describes has been a long time coming (and for once I can claim I partially predicted it—recall here and here). There are a number of major interlinked reasons for the current status quo (extreme referee shortage):
(i) an extreme scarcity of decent tenure track jobs relative to number of PhDs minted. This is a major source of anxiety throughout the profession
(ii) the unreasonably low acceptance rate of philosophy journals such that perfectly competent papers have to be refereed multiple times by a fixed number of referees
(iii) the perceived role that prestige and quality of publications play in hiring
and, perhaps,
(iv) the relative ease of free-riding in the current system. The real free riders are universities (who outsource quality control without rewarding it internally—my own university never even asks to list refereeing anywhere) rather than particular individuals.
In what follows, I take for granted that the low acceptance rates are functional, that is, maintained in order to produce prestige (recall this post). And the combination of (i-iii) has generated a veritable arms race in publication. In addition,
(v) the long delays in refereeing time, have made it prudent to have multiple papers under review at the same time, thus, exacerbating the underlying problem.
In what follows, I take for granted that the norm of regular publication has been good for professional philosophy. Reasonable people can disagree about this. But in my view it has allowed for more fine-grained discussion and for specializations to emerge that have improved quality. Yes, it risks generating hyperspecialized echo-chambers with lowest common denomination salami publications, but these, in turn, generate arbitrage opportunities for people who read widely or who practice what I call synthetic philosophy.
I also take for granted that (i) is entrenched in the political economy of modern universities. I am complicit in it; if you find a way to create a revolution, good for you. But what we can do, as a profession, is to cut the ties between prestige, journal publication, and hiring decisions and so reduce the need for referees. And we can do that if we destroy the role of pre-eminence of journals in manufacturing prestige, and restore them to the role of disseminators of knowledge and sites of discussion.
This can be achieved in two steps: first, all the major philosophy journals should commit to publishing, say, 25% of articles received. That’s considerably higher than the status quo (Philosophical Quarterly publishes just 4% of over 850 papers received). This will have six obvious effects: it will reduce the load on referees; it will end the excessive role of chance in where your paper is published; it will reduce (but not eliminate) the steep prestige hierarchy of our current journal system; it will reduce the excessive delays in getting work into print caused by the difficulty of finding referees; it will turn most major journals into paper mills; it will increase citation rates within the profession.
One may worry that once journals can’t play the role of extreme prestige generators, then it entrenches the roles of PhD granting departments. I agree, and that would be bad. So this is the second step: another institution has to take its place. Luckily, such a practice already exists in neighboring disciplines and is present in incipient form in ours: the article (and book) prize. What we need is a proliferation of prizes in philosophy: conference prizes (which are pre-publication); professional association prizes (of the major and smaller societies); best article prizes by journals themselves; and we need more versions of The Philosopher’s Annual. In fact, as Don Ainslie reminded me, there should be more prizes on different grounds too: philosophy has far too few of these, which in turn impacts success in interdisciplinary competitions (e.g. national scholarly societies, grant competitions, etc.).+
At this point one may worry that such prizes are easily shaped by politics, and that our time will be spent adjudicating prizes rather than refereeing. On the former, I don’t see how politics can be removed from the equation entirely (as it is, the journal system has a major ‘trust us’ component recall). I do not deny that genuine masked review, when it happens, is a real potential equalizer. And notice that on my proposal, we don’t eliminate that. People who come from non prestigious intellectual backgrounds can still use journal publication to become noticed. (But the initial signal’s power may well be reduced.) And one can use masked review also in prize committees. Over time, prizes that show a good track record in their judgment will become more prestigious, and so complement journals in prestige generation and quality control.
In a way, my proposal is a variant on, and complements, the post-peer crowd review advocated by Marcus Arvan, Liam Kofi Bright, and Remco Heesen which they have defended in print and at Daily Nous* and the ideas promoted by Katzav and Vaessen in a paper in Philosopher’s Imprint. (Recall also my post here during the Hypatia controvery, but I had limited the idea for work that might be controversial in various ways and in order to internalize political considerations in a transparent and fruitful way in the review process.) Using prizes is a mechanism to solve the coordination and collective action problem inherent in such proposals, and to assuage worries that non-experts may overwhelm the wisdom of our own crowds.
On the latter concern, about more time spent judging work for prizes, this is undeniably true. But I suspect it’s more fun to evaluate papers that have already been published—and so have undergone non-trivial quality and editorial control. Also, while writing a prize report is a non-trivial exercise, it’s only required for the winning paper(s). So, on balance I would expect it should reduce the load on those in the profession that are the back-bone of our current referee-system.**
+This comment was added after initial publication of the post at Digressions & Impressions.
*I also suspect my proposal solves an objection against those that think Condorcet jury theorem is not apt here, because prize awarding is standard setting not discovery of truth. But that’s for another time.
**I thank Helen de Cruz and Neil Levy for discussion.
Thanks for this. A really thoughtful post.
I’d like to place our crosshairs more directly on (iii), though: prestige. Yes, the current publication system is a poor prestige generator, and this needs to be fixed. The solutions you’ve proposed sound promising. But I worry they overlook the fact that prestige is a poor proxy for truth and knowledge, primarily because it is the handmaiden primarily of institutions, not of learning, and that for this reason our unreflective valorization of it lies at the heart of many problems in academic philosophy.
If that’s right, then whether we replace one prestige generator–pubs–with another–prizes (will shiny baubles be on offer, too?)–is only a cosmetic fix. What I think really needs to happen, but I doubt ever will, is that we in academia need to shake off our valorization of prestige, which I think in most cases really just serves as self-reinforcing ego stroking.
In its place, we come back to some basics: reading philosophy is fun (how many people here, when they read their first piece of philosophy, either in high school or in undergrad, fell in love with the ‘prestige’ of the author instead of how kinetic they found the ideas and arguments?); teaching is important; in our branch of education we do not produce primarily student products (‘outcomes’), but rather are in the business of slowing down, examining, teaching epistemic humility.
If we focus more on these things and less on how another’s prestige covertly affirms either our own or our desire for more, we’d have better pubs, a greatly improved referee system, and more time to play with the kids after dinner (unless doing so isn’t prestigious enough…).
Why am I skeptical this will happen? Primarily because of us: it is near impossible to go through a doctoral program without meeting a majority of faculty in our departments who are slavishly loyal to the siren song of prestige.
I think this comment misunderstands the role of prestige in a social system. If you have read a work several times, you can judge whether it was doing a good or bad job of whatever it is that it was aiming to do. But you don’t have to re-read it every single time that you want to decide how valuable it is. You can use your memory.
Prestige is just the institutional form of memory. If several specialists in my colleague’s field have already said that their writing is doing valuable work for them, and helping to advance their field, I can trust those specialists, and I don’t need to read the work myself and use my own in-expert judgment on that field. Saying that everything needs to be re-evaluated every time you comment is not a recipe for better publications and more time to play with the kids after dinner – it’s a recipe for more time spent slogging through re-reading work in fields other than your own that other people are better qualified to evaluate.
I’m wary of anything that lowers equal chances for prestige. It’s already virtually impossible to publish yourself “into another league”, so to say (at least as a grad student, but I think this holds across the board). Implementation of this proposal would only exacerbate that problem.
That said, I do like the prize idea as something to be done independently. In keeping with the spirit of my earlier remark, here’s one possible procedure for new prizes: anonymous pre-publication prizes. Editors from different journals (who don’t know the author), nominate a small number of accepted papers (perhaps in a certain subfield) to a jury on a rolling basis, which then uses some procedure (whether it be simple ranking or including a process of interactive deliberation) to elect a winner of the prize. If such a prize were given out twice a year, I’m confident most authors would be happy to wait with prepublication of their article (which would be necessary in order to maintain anonymity). (Twice a year is of course just a suggestion. But more often than twice a year would probably deflate the value of the prize too much, and less often might diminish the acceptability of waiting with prepublication too much.)
In all this I’m assuming that the discipline is, at this moment, highly prestige sensitive and is likely to stay so for the foreseeable future, as well as assuming that prestige comes with a host of benefits that we would do good to distribute fairly. Of course, one might think that we should diminish the importance of prestige in the first place, as John suggests above, but that’s another issue.
Not at all keen on this and there are alternatives:
get the major journals to cartel and require eg 2 referee reports from authors within 2 years forward & prior before a submission will be considered. low effort reports may be flagged by authors for review. authors who don’t follow through are sanctioned until they serve. exceptions can be made for those who have no pub/ref history. geniuses who are above ref work are invited to self publish.
But this creates a new loser: the reader.
Imagine a system where 5 articles and 2 books get awards in your subfield per year. OK, you’ll read those. What else? You can’t trust the journals — they’re paper mills. Your only options are (1) read less, (2) read famous people, and (3) read a bunch of rubbish. The result is that the field becomes (1) more desultory, (2) more unequal, or (3) more soul-crushing.
No thanks.
Does anyone seriously read every single article published in their field in respectable journals? Do people even read every single paper published in good journals on papers they’re writing? Of course not. Most of us do a fair amount of skimming and only read the good ones. Also, do you seriously think that accepting 1/4 of the papers submitted will turn the journals into paper mills? The average quality of papers submitted is not that low. Even if it did lower the quality somewhat, which I don’t think it will, to claim that it would turn them into paper mills assumes a pretty obvious false dichotomy.
Actually, yes. I read almost every single thing in my subfield – as long as it is in major journals/publishing houses. And I know a lot of people who do. It’s a subfield within Ancient philosophy (Plato/Aristotle M&E, etc), but I do keep up. I can do it, just barely, but I can – because it’s not that much actually. And it is not impossible because the publications get filtered by the journal peer-review and the collected volumes are usually easy to navigate. I dread all these suggestions to increase publications by journals. I get to review a lot and a lot of it is not worth publishing even if competently written. There is no need to publish that stuff. Personally, I do not believe all this is not such a huge problem as long as the discipline would manage somehow to reduce the number of submissions by grad students (which are generally easily recognizable, btw.) and standardize tenure track requirements so that they are reasonable.
Unless you’re in a tiny subfield no one reads every paper published. We all have our heuristics because that’s impossible. I take it that Joe and Platypus’s is something like read every paper in the journals that score well in Leiter’s polls, Brooks’s survey or other such rankings. That’s not mine nor do I think it’s a good one. The thing is that top journals already publish a good many turkeys and less prestigious journals publish a lot of interesting stuff. Honestly, when I think of the two best papers I read in the last year, Robillard and Strawser’s “On the Moral Exploitation of Soldiers” and Elgin’s “Fiction as Thought Experiment” I realize neither was published in a “top” philosophy journal.
No – I don’t care about Leiter’s ranking – what does that have to do with anything? There are specialist and generalist journals and they publish things by people from all over the world. If you can’t read what is being produced in your field and have no clue how to differentiate it, then what’s the point of it all? Why publish stuff nobody reads or even intends to read? If you cannot keep up with your field because it’s flooded with nonsense, then you should not be in favor of that becoming even worse.
Joe can you read French and German? And not just well enough to pass an exam with a dictionary in hand but well enough to read and understand academic articles? If so great. If not then it’s simply false that you read everything written in your field if it is as you say in ancient. You literally can’t. And did I ever say I have no clue how to differentiate it? I do. It might not be your way though I’m honestly not even sure what that is.
Yes, I can, as do many though perhaps not all people in my field (often it’s French or, rather than and, German, but many also do Italian). It’s part of what it is to be competent scholar in the field. There is also good stuff in Portuguese and Spanish, though that is less accessible (fortunately the people are publishing in English too). There is nothing unusual about this actually.
I said that I read papers in my area *if* they’re in top journals, not *only if*.
(1) If journals accepted 5x what they currently accept, then yes, they’d be paper mills. Most of their papers would be bad by current standards.
(2) Gotta agree with Joe. I have a few specialties, and I read every paper published in top journals about those specialties. It’s my job!
I don’t want a system where tons of bad things are published and, as a result, no one is expected to keep up with the journals.
I appreciate this suggestion, but perhaps it is desirable to first more firmly establish that we do in fact have a referee shortage. Ben Bradley’s post (https://dailynous.com/2022/02/18/is-peer-review-in-philosophy-broken-beyond-reasonable-repair/#comment-430821) makes me wonder whether the problem is a bit overblown. Also, the parallel discussion taking place over at PhilCocoon suggests that there are quite a number of junior people who are willing to review but are almost never asked. I’d be interested in proposals that would help bring these folks to the table.
Thank you for alerting us to Ben Bradley’s comment. I agree if the data he reports is the norm then the referee shortage problem is overblown.
I’m still thinking that some sort of “submissions cap” might be a nice solution. In Poland, we have a rule saying that only 4 publications per capita count when evaluating our institutions. Something similar, if implemented broadly, may do the trick.
I like these suggestions. Even if I didn’t agree with them, I like that they’re constructive instead of stupidly punitive unlike many other suggestions I’ve seen about how to respond to the crisis. And I do agree with them. I think they’d make a positive difference in peer review, and beyond that it is perverse that journals have been pressed into a role– prestige measuring– that they were never designed to fill, and a role, which, unlike disseminating knowledge, serves no useful function. I think the problems are deeper though. As long as there aren’t enough decent teaching jobs people will furiously try to publish their way into research jobs. If we had more jobs that were teaching focused which paid a living wage and had stability and benefits, then fewer people would be trying to claw their way into research jobs by throwing every piece of spaghetti they had at the wall and hoping some of it stuck. Another problem is that grad students get really bad advice from their placement directors that leads them to submit too many papers to top journals wasting everyone’s time. Publishing in Phil Review, while awesome, really isn’t going to get you a CC job or do much to get you a job at a teaching focused SLAC. Nor is it going to make you competitive for a job at an R1 if you go to a school that’s below about 25 in the Leiter rankings. Most grad students would be better off getting more teaching experience or just publishing in good but not top journals than they would buying as many tickets as they can for the top journal lottery and hoping one’s a winner.
One other thought/question: Is the situation as bad in other disciplines? Is part of the crisis due to philosophy’s weird focus on papers rather than books as the main research outputs?
Lots of disciplines in the Humanities struggle with scarcity of jobs and exploitation of adjuncts. But I believe philosophy journals have comparatively low acceptance rates
More prizes …
That is a rich suggestion – all the Yale, Harvard, Columbia, MIT, and Stanford grads will have just one or two more things to distinguish themselves from everyone else. Have you looked at who gets the prizes in the profession?!
Not if it will also cause more submissions. With significantly higher acceptance rates, the temptation of submitting papers that would not otherwise be submitted is greater.
Not necessarily a good thing, for reasons given in Kenny’s comment above.
The time it takes to find referees is relatively very short in comparison with the time it takes to get a paper from submission into print.
So everyone will have more citations, why should we care about this?
Not to mention that getting a research job will require a lot more publications, because with higher acceptance rates everyone looking for such a job will have more publications (which in turn will probably lead to more journal submissions, which will lead to greater referee overload, especially with artificially high acceptance rates …)
I think this is a brilliant proposal. Journals must commit to publishing 25% of received articles. Then we let citation counts, prizes, and so on, do further filtering. Yes.
It is worth holding in mind that not all journals have unlimited space in which to publish all good papers submitted to them: not all of their space is e-space. AJP’s online presence reflects its ‘paper presence’, with a contracted upper limit on the number of issue pages being published each year by the journal. Surely many journals are like that.
If this proposal were to compel journals to go all-digital, and thus relieve space pressures due to paper, I’d be all for it. I can’t understand why anybody would think paper journals are a good thing anymore.
I don’t know. When I look for papers that are relevant to the projects I’m working on, the journal of the paper has proved to be quite a reliable indicator of whether this paper was worth my time or not. Not a perfect indicator, to be sure! Some top journal papers are really not that good, and some relics are published in less prestigious journals. But overall, I have limited time, and I need to choose what to read to still make the deadlines. I wouldn’t want a useful indicator to go away, but this will most probably be what would happen if top journals will raise their acceptance rate 5 or 10 folds.
It sounds to me that the main problem is the lack of reviewers. The best way to fix that, I think, is by making reviewing a major part of what counts in our profession. Departments should require it and make it a part of their hiring and promoting decisions. Also, we might think of making it a custom to attach to every paper the major or most interesting parts where the paper was changed for the best due to the reviewers’ comments, and perhaps give the reviewers the option to identify themselves and take credit for that. I think Elife journal in biology does something similar. It seems that currently reviewing is a very thankless job.
Another idea that someone already proposed here is for journals to require a certain amount of reviewing by someone as a precondition for another publication in this journal.
I’m pretty sure .the problem became inevitable whenever it became considered normal to have “referees” in anything that was attempting to pass as philosophy. Reminds one of Wittgenstein, who seems to have thought it was all a game.
I read a shit load of articles, in philosophy and in other fields. There is no shortage of published papers. I also referee a lot – at least 180 papers + book 15? book manuscripts. Far too much unfinished crap is sent to journals (and publishers). So the idea that we are going to MAKE journals publish more is quite astounding. It makes me sick to my stomach
I propose the problem arises because nobody cares about philosophy papers except those who read philosophy papers. The result is that the discussion of peer review in philosophy is simply more of the same tightly circumscibed self-referential ism that led to the peer review “problem ” itself. If you now propose that “prizes” will solve this problem, then you are a reflection of the problem .
I don’t think there is a “referee crisis”. Although it certainly can be difficult to get referees sometimes and although that can be exasperating, nothing points at impending doom. Of course, any improvement is welcome. But from a practical point of view, how in the world can anyone hope to convince all those journals to accept 25% of all submissions if that is considerably more than what they are publishing now? They might not like it. Is there any practical way to get from here to there?
Doesn’t the problem lie as much in expectations to publish, win grants and the like?
I would rather we were trusted to publish at our own rates when we felt ready, and with zero pressure from departments or P&T committees. If that meant one top notch article in someone’s career, so be it. But I think most people would do more than that on pride alone.
I don’t see this happening given the influence of neoliberal managerialism on universities, but a reduction in the number of papers at a given time would be beneficial for refereeing, and, I surmise, would improve quality across the papers that are submittedl.
I worry that passing the buck to the ‘prize evaluators’ just kicks the can down the road. If prizes de facto replace journal publication as the gold standard currency for job-based evaluation, then there will be two things: an explosion of prizes offered, combined with mass confusion over what prizes are worth the most in job market currency. At least we all now know that Nous is hard to get in to, for example, harder than say Pacific Philosophical Quarterly. But an explosion of prizes on to the scene will leave people scratching their heads; moreover, ‘prize juries’ are potentially ripe for corruption to a greater extent than peer review is.
May I propose a very simple solution to the referee crisis (assuming there is one)? Assign only one referee to each submitted paper, rather than two or more. Only move to a second referee in case of serious uncertainty in the initial review. This immediately cuts the demand for referees in half. It should also speed up decisions in many cases.
My understanding is that this is the policy of Philosophical Studies. Makes sense! Wonder if the editors have encountered unexpected drawbacks.
This is, I think, an excellent idea. It would also solve the problem that many people complain of, namely that what one referee likes the other loathes so that article revisions have to meet contradictory constraints. Of course the unchecked caprices of mean-minded reviewers might be a problem, but even as things stand an adverse reaction from a mean-minded reviewer can often sink a paper even if the other referee praises it ot the skies. Under Michael Kremer’s suggested system if your paper meets with an unjust rejection, the reduction in turn-around times will make it easier to send it to the next journal where hopefully it will meet with a more favourable response. So I don’t think that the suggestion will make more likely that good papers will go unpublished. Will it make it more likely that BAD papers *will* be published? Perhaps. Happy-go-lucky sludge-tolerant referees will pass in papers that a second referee would have rightly rejected. So it isn’t a cost-free suggestion. But it will lead to a lot less unreadable garbage than Eric Schliesser’s proposal that journals should up their acceptance rates to 25%.