Two Recent Proposals for Fixing the Referee Crisis in Philosophy
Two philosophers have recently floated proposals for fixing the referee crisis in philosophy.
[photo by Justin Weinberg]
In the culmination of a four-part series on the problem at his blog—the last part is here, and includes links to the previous ones—Daniel Rubio (Toronto Metropolitan) focuses on what he identifies as “the inefficiency in editors seeking referees.”
Here’s how he describes the problem:
Most editors have a process like the following: after they read a paper, they ask people in their network who they think would be good. If no one bites, and they run out of suggested referees from their declines, they go to databases like PhilPapers (as suggested here) or look at authors in the paper’s footnotes to find more people. This makes it easy for people to fall through the cracks, or for editors to ask people at the wrong time. There has been a recent attempt at further improvement, in the form of a public spreadsheet where people can indicate their availability. But a google sheet is a clunky interface, and it relies on people knowing about it and regularly updating their status.
His recommendation is a database to which editors add minimal descriptions of papers that have passed desk review, and from which registered users can select papers to referee:
It would be better if available philosophers could volunteer to referee papers. It’s much easier for a journal (or journals) to maintain a viewable database of submissions than for the profession as a whole to maintain a database of available referees. And willing referees would not have to wait to be asked in order to contribute; they could simply find a paper that looked like a good fit and volunteer. This may not eliminate all of the inefficiencies in the referee search process, but it would take a big bite out of them.
Of course, the details matter. Tentatively, I suggest the following: once a paper has passed desk review at a journal, it is added to the database with a number, subdisciplines, and keyword tags from a standard menu such as the one PhilPapers offers. This will preserve anonymity and prevent any gaming of the system by friends/associates. Registered users (with some sort of screening criteria, as a first pass a Ph.D from a recognized institution) can then search by topic for papers within their specializations. They can then volunteer to referee a paper via a small form indicating their expected completion time-frame. This sends a message to the editor who posted it, who can then see if the volunteer fits their criteria for a referee for the paper. If the stars align, the referee receives the paper. If not, it stays up. Once the journal has enough referees lined up, the paper goes off the database.
This would make the search for referees much easier. But why would referees volunteer? Some may because of intrinsic features of the work, or in order to influence the development of the literature. But an additional incentive would be good. My suggestion: refereeing more papers should earn people more journal submissions. Perhaps by default, journals allow one submission per year. This will make room for people who cannot referee enough for legitimate reasons (e.g. they are too junior, doing other service work, or have various life circumstances such as illness or hardship). But in order to submit to the journal more than once per year, someone must generate enough referee reports for them to offset the ones they generate (plus some, to make up for the free submission everyone gets).
That’s one approach.
In a post at his blog, Richard Y. Chappell (Miami) says that the refereeing problem in philosophy stems from two facts:
(i) There is no personal cost incurred when you add to the reviewing burden (by submitting a paper to a journal); and
(ii) There is no reward for doing work (i.e., serving as a referee) that relieves the reviewing burden—nor for doing a good job of it.
In light of these facts, he proposes the following solution:
Journals should charge hefty submission fees, and use the money to pay editors and referees for their service.
This fixes the incentives instantly. It turns refereeing from a chore into an opportunity. And the desire to receive more such opportunities in future creates an incentive to do a good job. It also creates an incentive against flooding the system with low-quality submissions: if you had to pay, say, $300 per submission, you’re likely to be more discerning about which papers you send in.
Chappell anticipates some objections to his proposal. Here are two, but he covers more at his post.
“I can’t afford to pay that much to submit my papers to journals.”
Referee a few times and it should all balance out.
“Won’t monetary incentives encourage people to referee too much, and do a superficial, sloppy job of it?”
The hope would be that editors track the quality of referee reports, and use these track records to influence who they do or don’t invite to referee in future. (At worst, this may still be an improvement over the status quo, where it’s not uncommon for people to do a superficial, sloppy job while refereeing too little! )
Thoughts on these, readers? Anyone want to start implementing either of them?
Related:
Philosophers Available to Referee
A Little Rough Data about Journal Refereeing in Philosophy
How Editors Can Use PhilPeople to Find Referees
Flipping the System: One Possible Solution to the Publishing Odyssey
How to Fix the Referee Crisis in Professional Philosophy
How To Alleviate the Referee Crisis: A Proposal
How to Write a Referee Report
How do you handle revise and resubmit? If you get paid to review the paper a second time, you have a strong incentive to give an R&R decision. If you don’t get paid a second time, then you can just bow out for the second revision. This puts the editor in a difficult spot because now they have to find another reviewer who won’t get paid.
Yeah, that’s tricky. I suspect the best option would be for editors to communicate a norm that if you agree to referee (for whatever set price), there’s an expectation that you’ll see the process through to completion. If some life emergency occurs in the meantime and you have to bow out before the revision comes in, that’s understandable enough. But if you make a habit of dropping out early, the editors may lose trust in you and prefer to invite more reliable referees in future.
Or you could pay referees iff they see the paper through to completion.
This seems to incentivize a firm no – since the R/R will require more work.
I’d like to challenge a presupposition of these proposals. People keep saying ‘peer review is broken’ and all such things. Sure it’s not perfect for reasons many point out — but at the end of the day, the cream tends to rise to the top — stuff in the top five journals is pretty damn good, and beyond that, simply having a mechanism (top 5s etc) that allows people to compete for scares spots in prestigious journals via blind review.. helps with egalitarian meritocracy aims – even if you have a PhD from a crappy school, you can still have a great career in philosophy if you publish in top 5s! If we listen to some of these proposals that would undercut the current ‘gold standard’ value that these prestigious journal with minuscule acceptance rates provide in the competitive job market ecosystem, we end up making it harder for people without already prestigious networks, institutional affiliations to get jobs. the current system is great for allowing the possibility conditions for great philosophers to fight their way up and make a career. By opening up acceptance rates, etc, we are getting rid of this very thing that makes this possible. Let’s keep that gold standard and not mess with it! We need to keep, e.g.., top 5 prestige journals that reject most everything in order to allow an even playing field so anyone can make it in the profession. if you are from bob’s bible college and have no network at all, but you get some publications in the Phil review , nous, and jphil , mind, and ppr , you are going to get hired! The background that makes THAT possible is that everyone knows that it is super super hard to get anything in these journals. If the most prestigious journals had 30 percent acceptance rates, the person from bob’s bible college has no chance. you might counter and say that this person might write a great paper and you’d hire them on that basis. Uh huh. No one is going to read it! especially if we publish more and more stuff (as some of these proposals suggest). Let’s look for strategies for improving and tweaking, rather than replacing, our system.
A presupposition of this post is that there is one standard (“the cream”) that is shared by the entire discipline.
I don’t actually think that my papers in the “top-5” are the best of my work. They are the papers that best appeal to the people who tend to read and review for the “top-5.”
I guess I agree with the gist of the above. I mean, I agree with people who think it’s annoying and tough to feel like you need to publish top 5 to get a job (and at least some good top 10 generalist journals beyond that). But as the above post notes, it’s the toughness of doing that that serves a purpose — if you get something in Mind, for example, it kind of speaks for itself. This is obviously defeasible; some publications in top 5s are not great. But there is a benefit to having a presumptive marker like this, and if Mind let 30% of papers in the journal, it woudn’t be able to play this role any more. probably the right move is to just encourage referees to be more reliable, perhaps, by requiring reviews as a price for submission (or something like that!) where at least what we are doing is improving the current system rather than scrapping the very kinds of things that make it able to do what it can currently done.
Does it? I would’ve thought what should matter is whether what you wrote is insightful and contributes something to the field. Getting published in a “top journal” is at best only an indirect indicator of this. I’m not sure it’s best for us to outsource our assessment of the quality of work to journals on the basis of their prestige.
Mind did not publish a paper in aesthetics for almost forty years. The ones it did publish at either end of those nearly four decades are indeed good papers. But I’m not sure they’re the very best aesthetics papers submitted to Mind in that period. And I sincerely doubt they’re that much better than the best many they rejected in that period.
There are many contexts in which it is clearly not best to outsource our assessment of the quality of work to journals on the basis of prestige – for instance, in contexts where we have time and energy to devote to reading and evaluating the paper ourselves, and we have expertise in the field in question.
There are other contexts in which outsourcing assessment of the quality of work to a journal may be better than the available alternatives – for instance, in contexts where you are trying to assess the quality of hundreds of papers in just a few days (for instance, on the first pass reads of CVs of hundreds of applicants for a job), or in contexts where you have no relevant expertise (for instance, if you’re a dean reading a tenure file, and want something to corroborate what the tenure letters say).
Let’s say you need to assess the quality of hundreds of people in a few days. How did philosophers figure out that “Number of publications in the following list of specified prestigious journals” was the best heuristic for evaluating the competence and quality of philosophers? Did they carefully weigh this method of evaluation against other heuristics? Do we have empirical evidence supporting its efficacy? On what basis was this means of evaluation established?
And if you do use that method, it would be one thing if it were confined to those specific contexts. A necessary evil: sometimes we have to use heuristics. But “number of papers in top journals” becomes a target, an axis around which entire academic careers rotate. And when that happens, the measure is no longer merely a convenient tool for sorting through applicants; it becomes the goal of aspiring academics, and their success or failure determines whether they rise or fall or even remain in the field at all.
The stakes here are much higher than the mere need to review applicants quickly. The entire field gravitates around this metric. And rather than see philosophers seriously question whether this is a good thing and whether we need more fundamental changes, I see most people impatient or disinterested in questioning whether the problems philosophy faces are much more foundational than the lack of referees. I am not at all convinced that the need to quickly review applicants justifies placing so much of the weight of success on journal publications. People can be excellent lecturers, or mentors, or bloggers. People may write great textbooks, or syllabi, or be great debaters or produce amazing videos. Should we be downplaying all the many others ways someone may do good philosophy just because sorting through applicants without a convenient heuristic would be really cumbersome?
I haven’t noticed that aspiring academics take publication in top journals to be a final goal. It’s a good way of demonstrating, to administrators and faculty members from other disciplines especially, that one is publishing good and significant work. It’s hard to assess whether it’s better than other methods in the absence of any suggestions for what those other methods might be.
Publishing great textbooks or podcasts is also important! (And yes, even blogging!) And being a great teacher, even more so. It seems to me that The Profession does value those things, but whether they’re going to get someone a promotion depends on the standards of the person’s employer, and much less on what The Profession thinks.
I fully agree with Kenny that in many contexts it’s sensible to rely on the testimony of experts in evaluating someone’s work. (It’s odd to call this a ‘heuristic’.) His example of a tenure case makes this clear. It’s not just deans, either; I can’t competently evaluate the philosophical work of a Plato scholar or a philosopher of biology, so even if I do form my own judgment it would be arrogant or just foolish of me to ignore what experts in those fields think.
Whether aspiring academics are influenced by what’s in journals and adjust what they write, how they write, and so on isn’t something it would be easy to notice. There is plenty of evidence that in the sciences, researchers adjust what research projects they pursue on the basis of what they believe they can obtain funding for. Many researchers report that they spend little or no time studying what they want to study. Instead, they study what will be funded. When you set up an incentive system like “merit is based largely on publication in journals X, Y, Z,” it would be very surprising if this *didn’t* influence what people end up writing, especially those people that go on to be the most successful.
In any case, the exact impact of journal publication on the field is an empirical question. I see a lot of speculation (much of it my own, to be fair!) but little interest in empirically evaluating how our norms and institutions influence the field, much less what we might do about it.
My concern isn’t whether these are valued. Volunteering at a soup kitchen can be valued. But it probably isn’t going to land me a job at a top philosophy program. What I want to know his how much these sorts of things matter relative to publishing in a top journal. Are they given enough weight? Are journal publications given too much weight? If all we have here are vague sentiments and vibes, that doesn’t seem like a firm foundation for making policy decisions.
I never said that relying on the testimony of experts was a heuristic. The heuristic I described was using “Number of publications in the following list of specified prestigious journals” as a guide to evaluating competence and quality. Using the number of publications in certain journals as an indicator of someone’s competence is a paradigmatic case of employing a heuristic. There’s nothing odd about it at all.
I worry my concerns are not being recognized or addressed. I don’t disagree that, all else being equal, journal publications can be a useful and fast way to filter people who are more likely to be good hires than people with less impressive publications. But there are two problems with this (a) the quality of the filter and (b) Goodhart’s law. With respect to the former, I’m claiming that the peer review process itself, as it is presently implemented, isn’t that good of an indicator of quality. And second, I am arguing that broad knowledge of the metrics being employed alters the incentives of the field in ways that are detrimental due to Goodhart’s Law and other considerations.
With respect to Goodhart’s Law, if you use specific heuristics as criteria for hiring, then you create an incentive for people to work so as to do as well according to those metrics as possible. This turns the measure into a target, which can undermine its quality as a measure. In other words, you’ve created a Goodhart’s Law type situation.
And because status, prestige, and future success is based on these decisions, then it isn’t simply that reliance on publications as a heuristic for hiring assists you in making faster hiring decisions, it determines the entire future of all academics: which ones get hired, which ones get tenure, everything. Should the fate of all academic philosophers be decided mostly on the basis of the need to make quick and efficient hiring decisions?
Oh, I didn’t mean that emphasis placed on publishing in top journals has no influence on what people write. I meant I haven’t noticed that it’s a goal.
When you said it becomes a goal, did you mean only that it has some influence?
I see, well, I’m certain that the degree to which publishing articles matters in hiring varies enormously. There actually has been some smallish study of how much, as I recall, but I’d be interested in seeing more – although because of the huge variance I’m not sure how useful it would be.
Right, fair enough, it’s just that as far as I can tell you have no idea whether “number of publications in the following list of specified prestigious journals” is widely used as a heuristic. What we know is that publication is very widely taken, in certain decisions, to be an indicator of quality of a person’s work. To my mind, that’s just taking the testimony of experts into account. But maybe you just agree with this.
I think that publishing in top journals is a major goal of graduate students and early career professionals. It’s not their only goal. But I think that the fact that publishing in “top” journals is considered such a strong indication of the merits of philosophers that this has a significant influence on what people choose to study, what they choose to write, and who continues careers in academic philosophy. General attitudes about what philosophy is considered valuable or prestigious also influences writing samples, personal statements, where people apply to go to school, who chooses to stay in philosophy and who gives up and goes elsewhere, and so on.
Academic philosophy is a subculture. And the metrics it uses for hiring, admissions, and advancement all create incentives for what people do. This leads to, among other things, social reinforcement of whatever it is that is considered prestigious at the expense of what isn’t, even when that isn’t a good indicator of quality. It is probably also contributing to selection effects.
Speaking for myself, I often felt like a bit of a pariah because of my views and still do. I was often on the receiving end of explicitly discouraging remarks that could very well have motivated me to just quit.
Yea, I have no idea.
I don’t agree with that. I think the way papers are evaluated is far more complex and that it isn’t merely a matter of expert testimony. I think the professionalization of the field and the reliance on peer review very likely encourages people to write particular kinds of papers for particular kinds of people in their field and that the whole process, while it may serve as a filter for papers that are, e.g., free of error, well-written according to various criteria, and so on, is also selecting for papers that conform to various norms and stylistic conventions associated with “good” philosophy, norms and conventions that are likely narrow and that provide an advantage to people that approach philosophy in ways that are more generally accepted by mainstream analytic philosophers.
Consider my case: I’m extremely critical of reliance on intuitions and other aspects of contemporary analytic philosophy. I am unlikely to frame my arguments in ways that at least some analytic philosophers would find agreeable because I don’t share the same metaphilosophical outlook and dislike the methods and mainstream framings of problems. This puts me at a considerable disadvantage when it comes to publishing, not necessarily because my ideas aren’t worthwhile, but because I don’t share the methodological presuppositions of the people that are likely to be reviewing my work.
So while many philosophers may be experts in a relevant domain of knowledge, that expertise is often circumscribed to a narrow set of methods, presuppositions, norms, conventions, and ways of framing disputes. I think the result of this is that the current academic publishing system reinforces group think, conformity, and obedience to the status quo.
And I think the benefits of this approach are likely to be negligible. To be clear: I am not simply questioning whether the current peer review system in place, grafted as it is to academic journals, isn’t that good. I think it’s probably actively bad, and that we’d benefit if it were completely abandoned. I’ve seen little that would convince me otherwise. I’m certainly open good arguments and evidence for the value of the system we have in place. But so far, I haven’t seen great arguments for the value of the current institutions that are in place.
Fwiw, I totally feel you. Even if I’m not ready to follow your recommendations, I’m here for the diagnosis you’ve laid out in this thread.
Thank you. I notice the “thumbs up” ratio is not in my favor. Lots of people like responses to me. Not so many liking my comments. It’s reassuring to get some explicit positive feedback
Yeah no I see the things you see, and the way you’ve articulated your view of them has helped me see them better. I appreciate it.
I’m not sure how this comment is a criticism of the proposals from Rubio or Chappell. While some proposals for dealing with the referee crisis involve increasing acceptance rates, I don’t think either of the options offered here do. There is no reason why either of these proposals would require Mind or J Phil to alter their acceptance rates. Both proposals just make it easier to get referee reports, if they work as planned. So, this doesn’t seem like an objection to these proposals.
I want to express considerable skepticism for your remarks and to question some of your claims. In doing so, I don’t intend to convey antagonism but simply strong skepticism. If there is a good case to be made for the value of peer review, I’m open to it. But I simply haven’t seen it.
There are lots of ways for the “cream to rise the the top.” There’s no particularly good reason to think that the current way things are being done is very effective for achieving that outcome.
Academic publishing has a number of disadvantages. It’s slow and inefficient. It puts published work behind paywalls that are hard to access for people without institutional access. In practice, it typically lacks adequate transparency. Journals maintain arbitrary and tedious requirements for length, type of paper, and so on. It reinforces disciplinary and subdisciplinary divisions when greater interdisciplinary interaction would probably be a good thing. And, to my knowledge, there’s little evidence that peer review is an especially effective method for ensuring paper quality or serving as a good filter. Some of the issues with peer review may be more field-specific, but the concerns that e.g., Adam Mastroianni raises in The Rise and Fall of Peer Review are a good indication of where I stand on the status of peer review; I’ve seen little compelling reason to think that it’s actually benefited any of the fields in which its been implemented.
One of its biggest shortcomings is that It is vulnerable to Goodhart’s law: a measure that becomes a target ceases to be a good measure. When career advancement centers on publishing in “top” journals, whatever criteria these journals use becomes an incentive for people to write what’s publishable. In other words, people are rewarded not for writing something that is good in a truly open-ended way, but what’s good-according-to-journals. If this reward system is entrenched for long enough, people will begin to write for the rewards: the measure becomes the target.
Even if that were true, why think that’s a bad thing? Why should the current array of “prestigious journals” even be considered the “gold standard” for what should lead to career advancement?
Great relative to what counterfactual set of conditions? On what basis should we think the present system is great?
Regarding the allegedly egalitarian nature of journals: is it in fact true that they are egalitarian? How good is the blinding? Is there evidence that editors and reviewers are unable to reliably determine who authors are? Even if there is, can we be sure there aren’t stylistic and verbal indicators of those with a mainstream education that serves to signal institutional affiliation that could nevertheless bias editors or reviewers?
As far as it being hard to get into these journals: is that because it is a genuine indicator of quality? If so, how good of an indicator, and how do we know? I’m not asking these as rhetorical questions. Maybe there is good evidence for the value of peer review. And you may very well be correct about some or even all of your claims. But I have never, personally, seen a sustained and convincing attempt to actually provide good arguments and evidence for the value of peer review.
Lance In Philosophy, the top five are Phil Review, Nous, JPhil, PPR, Mind, with Phil Review top dog and the others debatable. After this, Phil Studies, Phil Imprint, Phil Quarterly, Australasian, Analysis, increasingly Ergo, then below that Synthese, Japa, APQ, PPQ, CJP. Ethics is a specialist journal with job market value similar to top-5. Anyway, we can spin wheels and whistle dixie all day long about about meta meta meta questions such as , whether any of the above really means anything if we want to really assess quality. What I’m talking about is job market heuristics , and career currency given those heuristics. For better or worse, given that job committees need to make decisions fast with increasingly big stacks of applicants, folks look to publications as a first defeasible bit of evidence. I say defeasible – maybe you read an applicant’s paper in Mind and think it’s bad. It can happen. But it remains that the first thing someone can do to put themselves in position to get a job/career advancement (aside from perhaps big fat grants , in certain countries like the EU) is land some papers in these journals, with preference for the the top 5, and at least top 10 – as these are things everyone knows is hard to do, very hard to do. Advising people otherwise is harmful to them; no body is going to read a paper in a crappy nowhere journal, sad but true. Sitting around with a wonderful paper in a bad journal , hoping someone will give it the time of day, will not a job secure.
That what you describe is how the field evaluates job candidates is exactly what I am objecting to. When you use criteria like publication in journals XYZ as a heuristic because you need a quick and convenient way to sort through a large applicant pool, that may seem like a great idea. But when people are aware of these sorts of metrics, it changes the way that they themselves approach philosophy. The measure becomes a goal, rather than an incidental byproduct of whatever it is a person would have written anyway. What you describe is precisely what leads to Goodhart’s Law being realized.
Then the entire field of academic philosophy pivots around the incentive system that you’ve created: success isn’t achieved by doing good philosophy, it’s achieved by getting as many papers in top journals as possible…which is used as a proxy for good philosophy. But when the proxy becomes the goal, this shifts what people do and how they do it away from what the original purpose: writing good philosophy. What was intended to be a simple heuristic for assessing job applicants becomes the raison d’être of the field, and rather than people publishing papers as a means to do philosophy, they do philosophy as a means to publish papers.
It’s completely backwards and wrongheaded. The entire field of philosophy shouldn’t be subordinate to the need for hiring committees to have easy ways to hire people.
Doesn’t this second proposal create a kind of catch-22 for new researchers?
Suppose you have 0 or very few publications. You are a graduate student.
Well, you won’t be selected as a reviewer— you haven’t proven yourself. But the purported offset for the prohibitively high cost of publishing is to review.
Thus, you’re at the earliest stage of your career, often amongst the lowest income of those trying to publish, and you’re given a new obstacle— one you may not be able to afford. Publish to become a reviewer, to offset costs, so you can publish.
Sure, some graduate students may have ample familial support but many certainly don’t.
I think universities would be obliged to support their graduate students by covering reasonable research expenses along these lines. (A small price to pay for so significantly improving the peer review system!)
Universities already rarely pay for graduate students to go to conferences, pay to conduct empirical research, or to do much else as it is. I find it unlikely they’d want to shell out even more money to cover publication costs.
I too worry about counting on university subsidies for grads. But you could also offer discounted rates (or no cost) for grad student submissions and for non-professional academic submissions. I have no idea what proportion of current journal submissions are graduate student or non-pro, but I expect you could still reap lots of the benefits of the Chappell proposal with these kinds of waivers or discounts.
Yea. Maybe. I’m not necessarily against it, and with some changes like that it might be more viable. Do journals really need these fees though? What kinds of profit margins do they have?
Richard’s idea was that the fees collected would be distributed to referees as payment for their services.
The profits of journals vary enormously. Elsevier journals make a lot of money. Most university presses operate most of their journals at a loss. And of course, open access journals like Imprint have to be subsidized.
Richard’s post suggests that departments may subsidize submissions by their graduate students. Of course, one may worry that this will come at the cost of lower stipends or reductions in other subsidies like travel support. But how high those penalties might be is an empirical question with no obvious a priori answer
It would also further advantage graduate students at departments who have the money to support students’ submissions
It doesn’t have an a priori answer, but there’s a lot of inductive evidence.
These subsidies are already rare.
These subsidies are already unevenly distributed.
Many universities will use any excuse to demand more and/or give less.
(e.g. raises leading to higher teaching loads)
So I’m not particularly optimistic.
I rather like some parts of Chappell’s proposal. But I think that it ignores a few structural problems with publishing and the profession. For example: some of us are stuck in TT or tenured jobs where we are horribly underpaid relative to the cost of living in the city we’re in (think: paid less than local K-12 teachers, and one’s pay raises never keep up with rapid local cost-of-living inflation… in my own case, I’m now paid less, in adjusted dollars, than when I started about 10 years ago, even though I’m tenured at and at the high end of our associate rank). Such jobs are almost always teaching-heavy roles, where one teaches 8+ courses a year, and publishing with a 4-4+ load is already difficult… yet the only way we have of trying to move to a better job is, of course, to publish well, because those hiring tend to care that one has published in good journals. However many of us also need to work side jobs on weekends or summers or whatever just to make up for the inadequate salary, in order to support our families adequately.
So on Chappell’s proposal, yes, I could earn some money by refereeing, and that’d be great… but I probably cannot afford to turn around and use that money to submit to a journal. I suspect something similar holds for most grad students and many early-career folks (VAPs, postdocs, etc.).
Since it’s a work expense, I’d hope that Universities would cover the submission costs for their faculty and grad students.
If some institutions failed to step up, and one wasn’t in a position to referee enough to balance out one’s submissions (most understandable for grad students, I think), Barry Lam commented with a nice suggestion involving crowdfunding via pre-print archives.
At the end of the day, I don’t think it’s always feasible to guarantee that an overall beneficial change will be beneficial for absolutely every individual. But if we imagine reversing the status quo, I think the case for abolishing pricing from the peer review system—in favor of the mess we have now—would be extremely weak.
> Since it’s a work expense, I’d hope that Universities would cover the submission costs for their faculty and grad students.
Doesn’t this defeat the purpose of the cost in the first place? The problem was that:
> There is no personal cost incurred when you add to the reviewing burden (by submitting a paper to a journal)
But if one’s institution covers the cost, then the cost isn’t personal. People will just submit as many things as they can (i.e. as they currently do) and pass the cost onto their institution.
There are a few ways around this, but I don’t see any good ones.
One might suggest that individuals must bear the cost themselves initially, and can be reimbursed down the line — so there is a personal cost (thus providing the disincentive we wanted) but it’s a cost of cash-flow, not of long term value. But this will worsen the problems for the TTs as is in the top comment we’re responding to, and for grad students mentioned above.
Or universities might have a fixed budget (in terms of cost, or in terms of number of submissions) for each department member. But this would just unfairly advantage those who could pay for extra submissions out of their own pocket (and thus disadvantage grad students and TTs).
Budgets aren’t unlimited; I assume universities would settle on some way to reasonably ration their subsidies. (This might involve a fixed quota for each member. Or it might involve some kind of selection process, e.g. needing faculty sign-off to approve a grad student paper as “journal-ready” and worth subsidizing. Or a mix of the two.)
I confess I don’t understand the normative perspective of those who would treat “unjust advantage” as a decisive consideration in this context. This isn’t a zero sum game (even if there are competitive elements to journals, some changes to the system may be net positive).
As I see it, the relevant question is whether or not the proposed system improves things compared to the status quo. For grad students specifically, you can ask: is it worth getting swift turnaround times (and perhaps higher-quality feedback), at the cost of limited subsidized submissions? The fact that some other (richer) people aren’t going to feel so limited when their subsidies run out just doesn’t seem so relevant.
As a conceptual point, if a policy change was better for everyone, but especially better for some, it would seem misguided to object to the change on the grounds that it unfairly “disadvantaged” those who benefited less from the change. Of course, I don’t claim that my proposal would be literally better for everyone—that isn’t a realistic goal. But I do want to encourage people to focus more on assessing the extent to which we should expect the change to benefit people relative to the status quo, rather than relative to those who would benefit most from the change. Benefiting less than some others is different from being harmed.
What’s wrong with sending a paper to the Journal of Philosophy and waiting a year or two?
We’re improving! Our oldest paper currently waiting for its first decision was submitted in early December. Now we have many great papers in queue for publication…
JPhil is going woke!
Hahaha shhhhh
Still – 5 months to first decision? One reason I’ve never submitted to J Phil.
Ah, but will it get comments? There’s no promise at five months, only after six.
We’ll post current stats on time to decision and rejects with comments on our website ASAP.
Nothing wrong with waiting 6 months at JPhil – that’s normal at the top flight places. better to wait six months or longer in the JPhil Lottery for initial verdict than to get ‘fast good news’ from a journal that doesnt’ have much cv value
On the second proposal: I’m not sure that there is any model for paying referees that isn’t likely to increase the stranglehold of the big publishers on scholarly publishing.
Since the admin involved in coordinating international payments is fairly onerous, generating staff and software costs, the only venues able to foot the startup costs, and the inevitable shortfalls that occur from time to time, will be the journals tied to the big publishers, incentivising people to referee for them over others. But while they may referee for the paying venues, they are likely to continue submitting to the free-to-submit ones. There might be more people in the system willing to referee, but they won’t be spread evenly, and they’ll congregate in the wrong places. So now we have efficient for-profit journals and inefficient non-profit journals. And the growing inefficiency of the non-profits vs the for-profits may ultimately deter authors from submitting to the smaller places, all to the benefit of the big five publishers.
The model also creates problematic incentives for editors. While some might welcome editors sending fewer papers out for review, I assume we’re all agreed that it’s better papers are desk rejected based on their quality and not how much money is in the coffers. (And an editor might send out for review papers that ought to have been desk rejected to funnel money to, e.g., one’s own grad students.)
Yes, anything involving money quickly becomes an administrative nightmare if it’s meant to involve people in more than one or two countries. I’ve mostly given up submitting to journals that involve payments because figuring out how to do that from India is a pain. The last time I submitted to Philosophers’ Imprint I was charged twice and I’m still waiting to hear from the University of Michigan about getting my money back. I’m close to just giving up and submitting another paper to the journal, since after all I paid for two, although I only got one receipt…
And that’s just getting my money to the journal. Getting money from some place is a whole separate rigamarole.
I have no doubt the fancy evil publishers have the process more or less streamlined, but as you point out, the end result will be strengthening those fancy evil publishers, which I’d like to avoid.
Another proposal for consideration I heard a while back: create some “journal bundles” where a few (maybe 3-5) journals are willing to use the same referee reports to evaluate an article. Then the “top” journal might reject it based on those reports, a mid-tier offers an R&R, and a lower-tier journal offers an accept. The author can then decide based on those options what they’d like to do. (My understanding is that law journals do something akin to this.)
Such a proposal has at least two distinct benefits as I see it: (1) reduces time on the author’s end, since you essentially get essentially 3-5 decisions in the amount of time it normally takes to get one, and (2) helps to alleviate the refereeing burden, since 2-3 referee reports can be used by multiple editors.
(Importantly, the idea would be to only have a *few* journals per bundle so that if the author is unhappy with the reports they still can send the paper somewhere else entirely and get new reports.)
This. If we combine this bundling idea with Daniel Rubio’s idea, the referee shortage could be improved. Improved enough for people to stop complaining? No. But improved enough for things to be noticeably better? Maybe, and at any rate, it’s worth a shot. (Admittedly, if I were a journal editor, I would not want my journal to be deemed “lower-tier” in a bundle. So maybe the categories could be “top-tier” and “mid-tier,” with no mention of “lower-tier.”)
I understand where Richard is coming from with the incentives problems and his proposed solution, but I don’t think it’s fair to require that large a payment for submission. Only rich schools will help with this payment. So everyone at non-rich schools will be disadvantaged. As for paying referees, I am all for that, provided the journals can actually afford it. Some can; some can’t. Is there a way we could pressure the ones that can afford it to do it?
In sum: Daniel’s idea + bundling + maybe pressuring the journals that can afford to pay referees to do it. Conjunctive solutions are inelegant but sometimes best!
The bundle system doesn’t need to have explicit tiers. It’ll just happen to be that people go with the journal they find most prestigious if they get accepted there.
I think we should be asking more fundamental questions about whether the entire system is worthwhile. People seem to want to patch a system that is of dubious value in the first place.
Should we be demanding papers go through mandatory pre-publication peer review to “count” towards career advancement in the first place? The whole thing seems slow, inefficient, facilitates gatekeeping, and, worst of all, is extremely vulnerable to Goodhart’s law. Maybe we should just do away with academic journals and peer review in their present form more or less in their entirety.
Here we go again.
I wish we were. I’d like to see some good objections to what I said and some good arguments in favor of the present system.
My concern is that paying referees to review papers changes the social meaning of refereeing in a problematic way. Right now refereeing is service to the profession — one does it out of a sense of duty. As soon as it’s a paid activity, people will start making the choice whether to referee based on whether they prefer to spend that amount of time to make that amount of money. This will predictably make a lot of people referee less, or not at all.
I believe that a study was done that showed that paying for blood donations both reduced supply and lowered the quality of the blood supply. That’s the kind of thing I’m worried about. (https://stanfordbloodcenter.org/pulse-volunteer-donations/)
The standard reply would be that donations would improve at some point once the payments were high enough. Also the psychological factors behind choosing to donate blood may not be identical to those behind choosing to review.
There is some tension here concerning what “the crisis” is supposed to be. One supposed crisis is that it is too hard to get referees for papers. But in some prior iteration of this discussion, I noted that the rate of accepting referee requests at Ergo was around 40%. To me, that does not suggest a crisis situation. A challenge maybe, but journals can handle it. I suggest we all just chill out a little bit about that issue.
In the first linked post in the OP (Neil Sinhababu’s post), the crisis seems to be that there is not enough room in journals to fit all the good philosophy that is getting produced. That problem cannot be solved by reducing numbers of submissions (by e.g. charging large submission fees); in fact it would exacerbate that problem. Ultimately this problem, if it is a problem, can be solved only by an influx of money to create new journals or expand the capacity of existing ones. (Or perhaps by a lowering of acceptable standards of copyediting and typesetting.)
Another crisis might be that some smallish subset of philosophers are having to do too much refereeing. This may be unfair to those people, relative to others in their situation who elect not to referee very much. Is this a crisis? I can’t bring myself to care too much about whether some other people are doing less work than me, but efforts to increase visibility of potential referees and spread the burden around seem like a good idea. Paying the referees, though, likely involves a net transfer of money from the financially worse-off to the better-off, given who is getting asked to take on the bulk of refereeing; that is not a great way to solve a fairness problem between members of the better-off group. (Unless universities are going to pay those fees; but they aren’t; plus that money could be put to better use, if the goal is to reduce unfairness.)
This is all really helpful and great points. I think you’re right that there is some overreaction. But the worst part I think is the time from submission to decision at many journals. Perhaps not a crisis but seriously bad anyway
If one were seriously minded to ensure payment for refereeing, the best course of action — insofar as it might actually work — would be to petition your institution to explicitly include refereeing in workload assessments, promotions, and so on. Universities, individually and in larger groups, frequently negotiate with publishers and are much better positioned to claw back at least some of this cost.
Richard Chappell has exactly the right diagnosis of the problem:
“(i) There is no personal cost incurred when you add to the reviewing burden (by submitting a paper to a journal); and
(ii) There is no reward for doing work (i.e., serving as a referee) that relieves the reviewing burden—nor for doing a good job of it.”
But paying for submissions with money is not the answer.
Notice, first, that there is still effectively no personal cost of submissions for people with significant familial wealth. So (i) is only partially addressed. And insofar as it is addressed, it is by deterring people with insufficient financial resources from submitting. This is an injustice not present in the current system, whatever its other failings.
Institutional funding is a speculative solution to that problem that’s partial at best. Even if, astonishingly, every philosophy department in the world provided a submission budget, wealthy people would still have the unfair advantage of resources beyond that budget.
A better solution is to solve the no cost/no reward problem by introducing a philosophy publishing currency in which costs and rewards are denominated. Daniel Rubio goes some way towards this. But if every journal allowed one free submission per year, I doubt there would be much incentive to acquire additional submission entitlements via reviewing. There would also be effectively no constraint on submissions, as there would be dozens of submission entitlements per person per year.
The best or least-bad proposal I’ve heard is the journal bundle system mentioned here by “Charging for submission” and by others previously. You have a group of journals at various levels of selectivity soliciting and sharing multiple reviews. They make their acceptance offers, if any, and the author takes their pick (or perhaps provides their preferences in advance).
How does this relate to the currency? If you had only one yearly free submission per journal bundle (rather than per journal) there would be rather more incentive to acquire additional submission entitlements via reviewing, addressing (ii). There would also be rather more constraint on submissions – as a first approximation, only as many submissions as bundles – addressing (i). A major bonus is that publishing would be much, much quicker as multiple journals would review each article simultaneously.
A concern one might have is that this favours established researchers who attract reviewing requests. One response is just to point out that they’re getting nothing for free: to get the benefit of extra submissions they would have to contribute to the reviews that make the entire system possible. But one might also introduce an element of disadvantage to being an in-demand reviewer, viz, that if you decline a review without good cause, you don’t only forego the additional submission but lose the free submission you would otherwise be due. In that respect junior researchers would be advantaged, having the guaranteed submission without having to work for it.
Another concern is that the most in-demand reviewers have many writing invitations too, so have little need of the submission currency. But I don’t see that offering something they don’t need would disincentivise them – especially as the thing being offered (submissions) doesn’t cheapen or monetise but is rather integral to the peer-review system.
There’s much to be ironed out, such as how many journals there should be per bundle. “Charging for submission” suggests only 3-5 so you have good alternatives should you not like a bundle’s reviews. I merely note that bundles with more journals would, ceteris paribus, be more likely to issue an acceptance from a journal the author wants to publish in and have the advantage of making their submission currency more valuable, further incentivising reviews and constraining submissions.
In any case, I think there is merit to this suggestion – and apologize for not recalling where it originated.
When should graduate students be permitted to be reviewers for journals? I think this question is fairly important, and yet I’ve seen very few direct answers to it. It’s an important question because, depending on the answer, it might expand the pool of reviewers to a degree that at least temporarily makes the current peer review system more bearable. Let’s say, arbitrarily, that 20% of grad students are fit to review. And of that group, let’s say, again arbitrarily, half of them review at least one paper during their time as a graduate student. 10% of the grad student population reviewing at least one paper would relieve a nice chunk of reviewing load. Informally surveying the people in my program over the years, it seems that less than 5% of graduate students ever review for a journal. Obviously this doesn’t answer any larger questions about how to better set up incentives, or whether reviewers should be monetarily compensated for their work, but it might be a nice band aid. Here’s the problem though: just like regular old reviewing, the amount of agreement about when students are fit to review is probably pretty low, and philosophers seem to be ok with deferring group action when there’s not enough agreement or a sufficient amount of disagreement. So my guess is that we’ll continue to largely not have grad students as reviewers, despite there being the possibility of a needed band aid in letting some subset of them review.
FWIW, as an editor over time at different venues I have never paid much attention to a person’s appointment status. I do look at whether they have published. A grad student with a paper I wind up knowing about that I think shows decent competence would be every bit as likely to get my very polite ask letter as a faculty member with a similar publication. Also, FWIW in response to a quote in the OP, I go to PhilPapers to find authors of good topically related papers if a name didn’t come to mind, long before I asked my friends for suggestions. If I recognize the person as someone I trust that’s a good prospect. If the abstract looks good it is as well.
While I am at it, I think the underlying problem is that we have incentives to publish too much too soon, but there is no way to change those incentives given the job markets. Papers have less in them and are narrower than they used to be, and this is just going to increase the number of papers that people try to publish when the number of publications is rewarded in various ways in hiring and in pay structures. Not everyone succumbs to those incentive structures but we now have to read a good bit more for the same amount of insight as people used to 40 years ago. (None of this is to dis presently active philosophers for responding to those incentives – my point is about the incentives.)
“I think the underlying problem is that we have incentives to publish too much too soon, but there is no way to change those incentives given the job markets. Papers have less in them and are narrower than they used to be, and this is just going to increase the number of papers that people try to publish when the number of publications is rewarded in various ways in hiring and in pay structures.”
Hear, hear! We could roll back recent developments. Journals would have to agree not to accept papers from grad students. They clog up the system. Their suitability for a job could be gauged by looking at (sections from) their PhD thesis – as used to be common practice. And as
Mark van Roojen said, papers are getting longer and narrower in scope. If one or two journals implement this policy, others may soon follow.
Everything you’re saying sounds good to me. It’s great that an editor doesn’t care about academic title alone (an editor that does seems to be doing something outright wrong). I also think that publish or perish has messed up the journal system and the average qualities of papers. But notice, you still use publication record to determine whether someone is fit to be a reviewer. Of course, no one is clamoring to be a reviewer, so increased reviewing demand isn’t even partially caused by the requirement of publication to be a reviewer mostly likely. But it does contract the pool of reviewers if publications are needed to review. I don’t know if you think that at least a publication should be required to get a non-adjunct job, but if you don’t think that should be a requirement, it’s unclear to me why you should think that having a publication should be a requirement to review. The best rationale I can think of for no pub req. for jobs but pub. req. for reviewing is that job search committees have time to look at dossiers in a way that editors don’t, and that examination is needed to figure out both if someone makes sense for a job and if someone is fit to be a reviewer. I haven’t been on a search committee, but I hear that the examination for dossiers is, on average, less thorough than it may seem to an outsider. All this to say that your answer to my original question — When should graduate students be permitted to be reviewers for journals? — is that they need a relevant publication. That’s not an unreasonable answer, but it might also not be the best answer if we’re, as a discipline, trying to better manage the reviewing load before larger changes to the journal system happen.
Short answer – shortcuts. When my department is hiring by the time someone is a finalist we are reading entire writing samples and somewhat judging for ourselves. I don’t have time to read whole writing samples of unpublished possible reviewers and even if I did I wouldn’t know where to get a sample. So I go to a list of abstracts (that’s what PhilPapers is) and find abstracts of papers close to the topic the paper that needs reviews is on, Then I either recognize a person who I already think is a good judge, or find an interesting abstract that looks competent. And ask its author.
A hiring decision is one thing, finding a referee is another and typically we have more time to devote to the first so we use fewer shortcuts. (As someone hired without any publications I would hate to think one needed one to be hired.)
While it would be better if editors had the time to do the extra research to vet unpublished referees, a tradeoff would have to be made for that time. I’d like to do a decent job teaching my classes, help my grad students finish their work and perhaps publish something. At least one of those would have to give if I had to vet unpublished referees.
That all makes sense. Although, I’d be curious to hear how often a zero pub. job candidate makes it to a shortlist. It seems clear to me that publications are the go-to metric for basically every kind of role assignment in (at least North American) philosophy. I think there’s a pretty straightforward way to figure out which grad students should be permitted to review that doesn’t take up editors’ time: a vouch system. Advisors vouch for grad students’ reviewer-level competency in some area, there’s public documentation of such vouches, and editors can look through those public vouches if they please. The public vouch system would also encourage grad students to publish ArXiv style (with abstracts!) so that prospective editors have something to chew on besides the vouch. (I tend to think more ArXiv-style publishing would be a good thing for a number of reasons). Under the vouch system — which integrates well with the first proposal by Daniel Rubio — the number of reviewers would go up and more grad students have incentive to publish separate from the journal. The only costs would be setting up and maintaining this public vouch system, and advisors having to make a judgement about when they should vouch for their students.
I like the first proposal. The main motive for checking the database and volunteering to review, for me at least, would be the chance to keep up more efficiently with new ideas in the fields that interest me. Of course, I already do a ton of reviewing of articles, but the first proposal (if all or most journals participated) would make it possible to cast a wider net.
I have said for some time now that it would be nice to have something like Dialectica’s fish pool, but for referees. I think opting in, at least for some journals even if not for all, would be a real boon. Waiting to be contacted seems pretty hit-and-miss.
I was not familiar with Dialectica’s fish pool system, it seems very cool. But it looks like they’ve encountered new problems, they’re not accepting submissions, volumes past 2020 (pub. 2023) only appear to be available as final proofs and volumes past 2024 don’t appear at all. Do you know what happened?
Here’s a different take: the referee crisis is, at heart, a citation crisis. Philosophers are notoriously stingy regarding their citation practices. It’s not great for the discipline to have super low impact factors. But it also makes it much harder to find referees. If a manuscript does a good job situating itself within the contemporary literature, an editor who is not an expert in whatever sub-topic should nonetheless easily be able to mine the references to find highly qualified reviewers. My understanding is that, in academia more broadly, this is a common way of finding reviewers. It’s also likely a less biased approach than having editors ply their personal networks.
Rather than looking at mining footnotes as a contingency plan, it ought to be an easy way to identify authors who are active in the contemporary debates and who are thus well situated to referee. But this only works if we widely cite our colleagues. If we only cite a handful of the most prestigious authors in our writing then we end up in a spot where making large databases, or charging graduate students to submit their manuscripts, begin to seem like reasonable solutions.
I’m somewhat skeptical that the first proposal (referees volunteering) would work better than what we have now. I do a fair amount of refereeing (between 10-15 papers a year for at least 10 years now) and don’t submit papers to journals that often (thee a year would be rare). I almost never say no, unless there’s a specific reason (such as that I’d already rejected a paper for another journal and it doesn’t seem much changed.) But, while I rarely say no, it’s hard for me to think that I’d search the work out on my own to anything like the same degree if doing so was left up to me. While Graham Harman, above, suggests that he might be more eager to review with such a set-up, it would be surprising to me if that was generally so, or at least if the number of people willing to do so without being asked was large enough to make up for the people who would not voluntarily seek out more work. I’d also be surprised if it found referees faster, and I can easily imagine papers that linger w/o a referee for a long time. Finally, I think the proposal underestimates the ability of people to try to game the system, if set up in the way described – telling their friends to check the pool for a new paper w/ certain key words over the next several days, and to exchange favorable referee reports and the like. I guess it would be interesting for some journal to give something like this a try, but for these reasons I’m skeptical that it will be an improvement.
I worry that some of the proposed solutions in the discussion would actually reduce the chance that weak or flawed papers are improved before they are published (the bundle of journals solution) – this is something the current system does help achieve.
I have published an article about the value of peer review in the sciences. See the attached:
https://direct.mit.edu/posc/article-abstract/33/2/202/127782/Is-the-Lack-of-Inter-referee-Reliability-a-Threat?redirectedFrom=fulltext
You can contact me if you cannot access it.
I don’t see how the bundle of journals (or journal consortium) approach would “reduce the chance that weak or flawed papers are improved before they are published … – this is something the current system does help achieve”.
What is the mechanism you envisage whereby articles are getting through this process unimproved?
To be clear, the consortium is commissioning more reviews than a single journal would – five or six, say. So it seems if anything there’s more feedback, this feedback comes much earlier in the process of placing a paper, and authors will have to make more revisions to accommodate more reviewers.
There should also be a greatly reduced chance of false positives (or negatives) as the judgments of one or two idiosyncratic reviewers would not be decisive.
The consortium idea never seems to get much traction in these discussions. As I see it it’s like a superpowered version of the current system, being faster, with more thorough and less fluky peer-review, and a balanced incentive structure. But I guess when others hear the idea they’re imagining something else, which loses benefits of the current system?
Thanks Carl,
I believe Springer is using such a model or mechanism now. As a referee I can opt to share my referee report for Synthese, for example, with other journal editors in the Springer/Nature family, if the author chooses to submit to one of their journals. So, my understanding is that a journal lower down on the scale can then decide to accept a paper based on existing reports (thus speeding things up, as so many people want). I have a good hunch that this is happening, because I see lots of papers I recommended be rejected from one Springer journal appear in another one, lower down the ranking almost in exactly the same shape I reviewed them (that is, with few or NO changes). I have reviewed many papers for Springer journals (roughly 78): 31 for Synthese, 13 for EJPS, 10 for Erkenntnis, 10 for Scientometrics, 3 for Phil Studies, 3 for JGPS, 3 for Science & Education, 2 for Found Chem., 2 for Found Science, and 1 for Philosophia.
Thanks for the reply, Brad.
It just seems common for lower-ranked journals to accept papers with only one positive review or possibly (if we’re talking very low) no positive reviews! That’s there regardless of whether the journal commissioned the reviews themselves or had them passed down from Synthese or Phil Stud.
To confirm that this issue isn’t inherent to consortia, consider that a consortium might only contain mid- to high-rank journals with strong editorial standards.
Thanks again Carl,
I think I may be missing something. Can you explain how a consortium of equally good journals would relieve any problems with the current system? My concern would be that if one of the journals do not want the paper, then there is a good chance none of them would want it. Then the author would have to find a lower tier consortium. So I do not see how the existence of consortia (is that a word?!) have given us anything.
Not “equally good” but good (mid or high tier). Just because Mind or Nous reject something doesn’t mean another good journal won’t pick it up, much as at present.
The advantage is that one round of multiple, shared reviews makes the whole process much more efficient and less random. I concede that many people find this counterintuitive.
There is a good chance that AI referees will be abundant and comparable to human expert performance within 2 years
Yes, not to mention less biased and more polite.
I will again post a suggestion that I made in a comment on a post here several years ago on this same topic (I have since seen others make it independently). It at least cuts the problem in half.
Make the standard be one referee per paper. Only call on a second referee if something is off about the first report.
Why isn’t this the norm? I suppose that in journals wherein the standards are clearer and more objective (in some sense) — e.g., logic-ish papers etc. — than in other areas, the MKremer suggestion is natural if not default. But why isn’t this the norm in *at least* specialty journals more generally (e.g., Ethics, History of X, Whatever-ism, etc.)? Genuine question. Maybe so-called generalist journals (maybe Mind/JP/AJP/Nous) reflect that two referees somehow improve the verdict to publish. I don’t know. But specialist journals wouldn’t be unreasonable in having referee practices that reflect the specific discipline and standards of expertise. I think. In any event, Kremer’s proposal has the virtues he notes, but it also raises the question above: why isn’t this already the default? (I’m asking a question. I’m not offering an argument. I actually agree with Ben Bradley above on a lot of this stuff.)
JC, here are two reasons to have two reviewers.
But both defaults are reasonable. I don’t mean to say it’s unreasonable to go with a single reviewer.
Two reviewers is definitely better than one, and reduces noise, but it’s unlikely to reduce it by enough to make it a very reliable process to rely on two reviewers. Ideally papers would be reviewed by many more than two people.
Jamie – thanks for a clear and interesting answer.
Reason (2) gives me pause. My experience suggests that, even when rules of debate are clear and fixed, any given topic has more than two “major sides” (unless you just mean p and its logical negation, and assume classical constraints, where p is the “topic” etc.). If a single referee is competent, professional (aims to be dispassionate, fair, etc.), and — what should be a given in philosophy — ably considers all relevant “sides”, I don’t see that reason (2) really pushes for more than one referee and the editor’s (or, as it may be, editors’) considered judgement. Of course, you’re just pointing to defaults, but I wonder whether (2) is a useful default.
It sounds like I’m offering an argument against two or more referees covering all relevant sides. I’m not. I’m just genuinely puzzled by the question that Michael Kremer’s proposal raises.
Lest it be unclear, while I think that the current journal activity is in pretty bad shape, I am *very* grateful to all editors and competent, professional, fair referees without whom our profession would be in absolutely terrible shape (which it isn’t).
I have zero expertise here but I suspect relying on one reviewer would lead to a higher acceptance rate or to more editorial decisions to reject papers that reviewers suggest be accepted, and I think a lot of journals would not like either of those results.
(It feels to me like each reviewer is basically a coin flip, so my chances of getting 1 yes verdict are 50%, whereas my chances of getting 2 yes verdicts are 50% x 50%, i.e. 25%. So that is why I think one reviewer would lead to a higher acceptance rate, unless journals start to ignore reviewers more often.)
I don’t mean to spam this, but here is my fleshing out of this proposal—I remember your kind recognition of the proposal the last time I posted it in a comment thread on DailyNous, Michael.
As to the question as to why this isn’t the default (recall that the default is a minimum of one reviewer rather than two, suitably complicated papers may require two or more reviewers in any case), my theory would be that philosophers simply overestimate the risk of degraded quality control. I make some remarks on that objection in the post.
Two concerns about the payment model:
First, unless something changes dramatically, this fix promises faster turnaround time on papers but not higher acceptance rates. It’s hard enough rolling the dice at a journal with a 4% acceptance rate, but now I have to pay a bunch too? And this idea of “Well, maybe carefully reflect on the quality of the paper and send it to an appropriate place” is a fantasy. All of the top thirty journals are pretty darn good and surely often reject pretty good papers that will land somewhere. And don’t I deserve to dream big?
Second, I’ll just harp on Daniel Weltman’s point to say that we shouldn’t underestimate the administrative burden of securing reimbursement from a dozen different journals across country lines. I’m not sure how high the payments would have to be for it to appropriately incentivize me to contribute, and so it’s more likely to act as an incentive for early career scholars that perhaps should do less reviewing and admin.
Friendly amendment: I’ve seen some journals in other disciplines have fees to submit that you get back if you get rejected. Even if we got back 80% of it, maybe that’d be enough to fund the process without making me go broke to fund my dreams.
Chappell’s diagnosis of the problem hits the nail on the head, but I don’t think that having submission fees and paying referees is the way to go (for many of the reasons others have mentioned in the threat).
But, insofar as problem ii) is concerned, there is an easier fix, I think. Philosophers, like everybody else, are motivated by money, of course. But acedemia is first and foremost a credit economy: what counts and motivates a lot of us is being esteemed and being well-regarded by esteemed and well-regarded people in the field. (A substantial reason why that sort of credit is important is because *that* is what gets you the jobs. That’s what publishing in prestigious journals is a proxy for, which is the root of the whole problem, after all).
So, the way to solve the problem is to reward referees with extra credit. Credit is not a resource that we need government grants to obtain and distribute, and it doesn’t require bypassing financial and bureacuratic hurdles to distribute, so it’s cheaper and easy to manipulate.
One easy way is to make refereeing count towards one tenure, or being part of the workload (refereeing n papers can get you a lighter admin or teaching load, e.g.). Of course, that is something that university administrators have to implement, so it’s not just up to the community of philosophers. But so would all the other incentives, and I have an easier time picturing a dean being convinced to allow one to forego a couple of meetings as erasmus coordinator for having refereed 20 papers the year before than forking out a lot of money for that same person to submit a paper that will probably be rejected to a journal.
But more simply, we could start caring more about refereeing, collectively –– making it into a credit factor that reveals the value and contribution of a person to the field, including but not limited to when we make hiring decisions.
Like, it should be a line in your CV that in 2025 you refereed n papers, and we collectively could start caring about it at least a little bit, and it should be something that is discussed in hiring committees, alongside one’s publications or teaching experience.
Of course, if this sort of credit becomes relevant in hiring practices, it would disproportionately impact those who are on the job market or that haven’t got tenure yet, so it’s an imperfect solution: it doesn’t give Big Shot many reasons to referee more. But then again, would extra £150?
Volunteering is the best option and solution. Alternative, if palatable and acceptable lies in AI.
When my kids were little, we were part of a neighborhood babysitting co-op with other parents of small kids. You’d earn points for babysitting, and could spend those points having someone else in the co-op babysit for your kids. If you wanted a night out, you needed to provide that service for other parents in the co-op. No money could be exchanged. It was just trading services. Seems like this would work for journals and refereeing; just need a way to keep track of credits. Maybe a problem of scale, but the philosophical community is full of smart people who surely could tackle this kind of issue. Heck, maybe there’d even be a journal article in it. 😉
Heck, maybe there’d even be a journal article in it 😉
Write it well enough and maybe even a nobel prize!
Assuming they are also authors, referees are paid, in the labour of the referees and editors who handle their papers.
In this thread: some Americans forgetting that other countries exist, some faculty forgetting graduate students exist, and some Global North academics forgetting that the Global South exists. Not to mention proposals that are curiously naive about perverse incentives.
Perhaps worth considering each of these in turn before suggesting more financial barriers to academic debates..