A Public Database of Referee Service (guest post)
A few years ago, Neil Sinhababu, associate professor of philosophy at the National University of Singapore, wrote about the “publication crisis” in academic philosophy in a post entitled “2,000 Spaces for 10,000 Papers: Why Everything Gets Rejected & Referees Are Exhausted.” In this guest post*, he follows up with a proposal for how to help make things better.
[David Moreno – untitled (detail)]
A Public Database of Referee Service
by Neil Sinhababu
Peer review of journal articles is a central but increasingly dysfunctional part of professional philosophy. Authors wait months for referee reports, often receiving rejections without comments. Editors implore referees to review manuscripts, often without success. Referees work hard to evaluate other philosophers’ manuscripts, almost invariably without reward.
To publish good peer-reviewed philosophy, editors need good referees and good authors. But while authors enjoy the spotlight, referees work in darkness. Because authorship is rewarded while refereeing isn’t, there are too many manuscripts to properly referee. This would change if journal editors could publicly credit referees in a way that administrators could recognize. This could be achieved simply by making public how much refereeing everyone had done.
Meaningful Benefit for Diligent Referees
Research universities seek to employ experts who improve their academic reputation. Being asked to referee is a sign of one’s expertise, and writing good reports is the sort of professional service that university promotion and tenure systems are designed to reward. With a public record of how many papers each philosopher has refereed, refereeing statistics might figure in evaluating candidates for senior research positions. Writers of external letters for senior research positions might find this information especially useful. The database will distinguish diligent and highly regarded scholars from those who refuse to referee or whom editors quietly recognize as unhelpful.
Administrators from outside philosophy often have final say over whether philosophers are hired or promoted. As nonphilosophers, they can’t assess philosophical research on the merits. Quantitative measures of expertise and service keep them from making erratic decisions when philosophers’ careers are at stake. Serving on a committee that evaluates nonphilosophers for tenure has made it clear to me how much administrators want information of this form. By making it publicly known how many papers everyone has refereed, journal editors can direct administrators to reward those who do more than their fair share.
The lack of any meaningful credit for refereeing threatens the system of blind review. Heavily rewarding successful authorship has generated an oversupply of papers relative to journal space. This oversupply of papers is mathematically expressed in high rejection rates, and the way authors respond to it makes it worse. Those who need fast professional credit for jobs or tenure must put even more papers into the system in hopes of getting something published. As the oversupply of papers overwhelms refereeing capacity, desk-rejection becomes increasingly common, and papers must be both written to appease referees and to avoid desk-rejection at first glance. Simply writing good papers for blind-reviewed journals becomes a less viable path for authors, and cultivating personal connections to those who control non-blinded venues becomes comparatively more viable.
Reward for refereeing would allow the drive towards professional advancement to express itself in refereeing and not just authorship. This helps to maintain the sustainability of blind review in both ways. First, it increases the supply of referees. Second, it offers those who need professional advancement an alternative to exacerbating the oversupply of papers. A public database of refereeing, consulted by university administrators, is the sort of academic infrastructure that would have these effects.
Settings and Features of the Database
Many different types of statistics might be recorded on the database. Publicizing helpful behavior might encourage more people to do helpful things.
For example, my colleague Zach Barnett suggests publicizing how quickly referees send reports. This would likely lead to faster refereeing of papers, with referees striving to distinguish themselves as especially prompt. While any incentives for refereeing would strengthen the hands of journal editors to insist that referees generally do their jobs well, publicizing how quickly people refereed would speed things up further. Other forms of information might be included to beneficial effect, and innovative thinkers should explore the possibilities.
Protecting blind review requires excluding information that might connect individual referees with specific reports. Providing too much information might enable authors to identify referees who rejected their papers. There are many ways to prevent this by excluding or obscuring information. Refereeing statistics might be updated only once every few months or even annually, so that nobody can tell who rejected a paper at a specific time. The identities of the journals that someone refereed for might be stated only in some general way, or excluded entirely. Sensitive information might be restricted only to those with special authorizations or permissions.
Journal editors will likely have final say about which features the database has (and whether it comes into existence). To avoid requiring excessive additional labor, it will have to be integrated deeply and automatically into journal refereeing systems. This isn’t possible without the active participation of editors, who will therefore collectively determine whether and how a database is set up. This is a favorable alignment of interests. Editors need to solve their problem of having too few referees. By setting up the database, they can solve their problem together with everyone else’s problems.
Creating the Database
The closest existing thing to such a database is Publons, which I signed up for years ago hoping that it might play the role in the profession envisioned here. But it hasn’t gained widespread acceptance and seems unlikely to do so. Its current owners have different priorities than philosophers.
Publons was acquired in 2017 by Clarivate (NYSE: CLVT), formed in 2016 when a private equity firm and an investment manager bought an intellectual property business. Valued at $17 billion, Clarivate has a “bold entrepreneurial mission to help customers reduce the time from new ideas to life-changing innovation.” Whether the mission of philosophy is to discover the truth as traditionalists claim or to change the world as Marxists claim, a bold entrepreneurial mission is unlikely to remain aligned with it. By charging wealthy institutional customers large sums to access information while denying access to philosophers who can’t afford the fees, bold entrepreneurs can profit at the expense of philosophy. To avoid being overcharged for information they generated, philosophers should strike to keep it in the hands of public or philosophical entities like PhilPapers rather than private businesses. Nevertheless, the fact that so many journals have integrated their systems with Publons reveals the interest in publicizing refereeing data.
Integration with ORCID is a more promising possibility. After academic publishers established ORCID to better match author names to journal articles in 2009, they put it in the hands of an independent nonprofit group, which has controlled it since 2010. It’s not clear to me whether ORCID could be integrated with a database of this sort. But it has over 10 million users, so it’s accepted widely enough to be part of a solution.
Whether and how a refereeing database takes shape ultimately depends on journal editors. As the problem of too many manuscripts and too few referees becomes increasingly severe, editors’ need for a systemic solution increases. Setting up academic infrastructure to publicly credit referees for their service would rebalance the incentives and solve the problem. There are many details to work out, for example regarding how the database should treat differences in the journals’ refereeing procedures. But if editors need referees badly enough, they’ll have motivation to negotiate creative solutions when required.
Together, journal editors could solve systemic problems with peer review. They collectively have the power to reorder the system as they please. Professional philosophy dances to their tune, and if they change the tune, they’ll change the dance. Setting up a database that enables administrators to reward diligent referees would solve their problems, authors’ problems, administrators’ problems, and referees’ problems all at once.
Related: “The Publication Emergency” by J. David Velleman, “A Plea for More Short Journal Publications” by Avram Hiller, “Flipping the System: One Possible Solution to the Publishing Odyssey” by Felix Bender, “How to Write a Referee Report” by John Greco, “Reforming Refereeing” by Aaron Garrett, “The ‘Insanely Low Acceptance Rates’ of Philosophy Journals“, “The BJPS Referee Of The Year Award“, “How to Accelerate Refereeing“, “Citing (and Thanking) the Referees at the Journal that Rejected You, Part 2“, “Should Publishers Pay Referees and Authors?“, “Advice on Refereeing Papers“.
To be fair to me, the suggestion to publicize reviewers’ average turnaround times was made casually and spontaneously. I didn’t give it serious thought, and it certainly wasn’t something I expected to appear on Daily Nous.
For what it’s worth, if there were a ‘leaderboard’ of review times, I would be toward the lower end of the list. In fact, I am overdue on two reports right now.
Sorry, Zach! I told you I’d put it in when you said that back in April. But I say enough wacky stuff that you probably didn’t take it that seriously.
No worries at all Neil. I remember that conversation. I just didn’t know I’d be named as someone who agrees/wants to take things a step further. This was a good discussion though.
For what it’s worth, I finished my overdue reports.
I hate everything about this.
Do you hate getting your papers refereed instead of desk-rejected?
In the last, say, ten instances: Yes. Actually, the last dozen or so referee reports I’ve gotten were worse than just being desk rejected.
I don’t like the status quo. But to someone like me who has severe anxiety and depression that is routinely set off by referee work, the proposal here looks like an absolute fucking nightmare.
While I understand this reply, it is an unfortunate consequence of the proposal in question that it glorifies the free labor we perform to help make Elsevier (etc.) rich.
I would certainly be in favor of glorification of the labor we perform for non-profit, open-source journals.
Gamification of professional service? Good luck with that.
At least in the UK, professional service is *already* gamified as part of promotion criteria. For example: in order to make it from lecturer (grade 8) to senior lecturer (grade 9), you need to meet criteria that include hitting benchmarks for (among other things) (i) grant income averages per yer (needs to be above a certain number, e.g., £15000 per year) and (ii) PGR students supervised (e.g., over 2.5 students per year). The above numbers are just estimates – I can’t remember the exact figures. The point is: there is no similar benchmark for referee work factored into promotion criteria in the UK. Of course, one can (fairly probably!) criticise the entire UK promotion structure. However, given that it is in place as it is, it seems like adding a similar benchmark (e.g., 10 papers refereed per year) which can be evidenced would be a nice way to, at the same time (i) ensure that this important service to the profession isn’t overlooked in pay-based recognition; and (ii) to discourage freeloading on the referee system.
Thanks, Micah. I’d add that every way of measuring professional achievement can look like gamification. If credit for refereeing is gamification, we might take authorship to have been successfully gamified, with wins at the game recorded as names of journals on one’s CV.
I will accept your point and raise you one further. I think authorship already is gamified and that the rules for this game are pretty well established. Here is the game I take myself to be playing. Getting in a decent journal (e.g., at least something with 90% rejection rate) is like winning a regular season event. There are five ‘majors’ – Phil Review, Nous, JPhil, PPR, Mind. Your greatness in the game is largely a measure of how many majors you’ve won. I would rather have a career where I’ve won 15 majors and only three other tournaments, than a career where I’ve won 80 tournaments and only 1 major. Finally: winning all five majors is called the ‘career grand slam’ – very rare actually. And the ‘grand slam’ (unheard of!) requires publishing in all five venues in one year. It would be interesting to see what would happen if Williamson tried to get a grand slam.
I don’t see how this is supposed to help with the problem it aims to solve.
Let’s imagine an extreme case of succeeding at lowering turnaround times. Every journal gets a verdict back to every author in 2 weeks. What happens next?
Well, now there is no reason to try and aim your paper at a journal at the right level for it. May as well send every thought you have to Phil Review, then 2 weeks later to Mind, then 2 weeks later to Nous or Ethics or whatever. And if you work the system well, and have a bit of luck, even a not-so-great paper will be in a reasonably good journal in 6-9 months.
But this couldn’t really be an equilibrium, because in this situation every single paper gets sent in the first instance to Phil Review. And the folks at Cornell do an amazing job, but they can’t process 10,000 papers/year – no matter how good a leaderboard of referee times we have. And do what you want with the referee system, if *every* journal gets most of these 10000 papers/year (as each author works their way one-by-one down the prestige hierarchy), the refereeing load will be unbearable.
That’s what happens in the limit, but I think the thought experiment shows what happens in more realistic situations. The better the review system gets, the more work it will have to do. Now there are reasons to make it better, because at some point it becomes a detriment to good scholarship. But some of these proposals sound to me like fixing the traffic problem by adding lanes to the highway; you’re just going to encourage more traffic and pretty soon the problem will be back.
Personally I think the root of the problem doesn’t entirely lie with the journals – it also lies with search committees and tenure committees. There are plenty of journals out there: over 250 just on DOAJ. There is no “oversupply of papers relative to space”. There is an oversupply of papers relative to the space that search committees and tenure committees have decided that they care about.
If search and tenure committees (a) cared about quality more than quantity, and (b) cared about quality more than venue, we would have a different (and more manageable) set of problems. But because search committees and tenure committees keep outsourcing their decision making to the editors of Phil Review, Ethics, Mind etc, rather than doing the job they are paid to do, we keep coming up with plans like this that just move the bump in the carpet around.
I agree with almost all of Brian’s post.
I reject only the analogy to road traffic issues — more lanes addresses that issue effectively in a pretty wide set of circumstances.
Working within Brian’s analogy, the database turns excess cars into more lanes. So it’s not just adding lanes – it’s reducing cars.
I’m not entirely sure I agree with this point. Search committees and tenure committees aren’t paid any more than journal referees (in a sense, I think Neil’s proposal is to make journal refereeing a more integrated part of the job, more comparable to internal service on these committees). And I also think that if we as a profession care about quality, then it would be better to get our more public-facing organs, like journals, aligned to quality in a way that private-facing ones like hiring committees can then piggy-back on, rather than getting the private-facing organs to each run their own independent quality checks at their own cost and leave the journals in a tragedy of the commons.
That said, the induced demand on highway lanes is a great analogy here.
Why aren’t search and tenure committees paid? It’s part of the service component of my job, done on my employer’s time. (But then, so’s refereeing). The issue is more that not all aspects of the job are equally recognized in professional advancement.
I repeat my standard suggestion that we need more specialization among top journals. If everyone sent their best ethics papers to the top ethics journals and then worked down the ethics journals, and mutatis mutandis for other subjects, the crowding gets significantly alleviated.
I agree, except I’d put the emphasis on the other side of the equation. We need more specialized journals among the top journals. That is, we all need to start treating the existing and very good specialized journals as the top journals they are.
That can work (it already does work in philosophy of science to a large extent).
That said, having an arrangement by (say) Mind, JPhil and Phil Review where they divide out subject areas among themselves (and maybe rotate every 3 years) would also work.
David, how do you make sure there are enough referees at the ethics journals? A system-wide lack of referees is the problem that the database is supposed to solve.
The thought is that if papers go directly to specialist journals, they will get published with fewer rejections, and so fewer referees per paper will be needed.
That thought is premised on specialist journals being easier to publish in because they’re less prestigious, and therefore having higher acceptance rates.
Right now Ethics is desk-rejecting half its submissions, and accepting only 4%. It looks like they’re feeling the refereeing crunch as much as anybody. As soon as a specialty journal becomes prestigious like the top general-interest journals, the problem regenerates itself there, because prestigious venues get swamped with papers.
https://www.journals.uchicago.edu/journals/et/after-submission
You can tell authors to stop submitting to such prestigious journals altogether, but unless you give them some other way to get professional credit, they’re going to ignore you and pursue their career interests. The rewards of publishing in top journals are high enough that playing the lottery there is the best move for many people. The idea of the database is to offer credit for something that isn’t even a form of authorship, namely refereeing.
It’s not really premised on that.
A stylized model: suppose that we have 3 generalist journals with an agreed rank-order of prestige. Everyone submits their papers first to journal 1, then if rejected, to journal 2, and then if rejected, to journal 3.
If instead we had 3 top-prestige journals specialized so as to cover a third of the discipline, everyone would send their papers to one of them, and then on to other journals if they didn’t get into it.
On the second model, each paper gets refereed once by those three journals. On the first, it gets refereed up to three times.
That merely shifts the burden of refereeing labor from the 3 top-prestige journals to the “other journals”. The same total amount of refereeing is needed; it just isn’t going on at the 3 journals you’re focusing on at the beginning.
No it doesn’t. Suppose 30% of papers get accepted in the 3 top journals. Then in the generalist model, a submitted paper gets refereed on average 2.7 times; in the specialist model, it gets refereed once. At the end of that process, the remaining 70% of papers need to be considered by the other journals either way. But at this stage in the process we have a reduction of 1.7 lots of refereeing per paper. That’s an actual reduction, not just a reallocation. (And potentially there are further reductions of similar kind in the ‘other journals’, though I’m less sure how to model that.)
This still looks like an overall reallocation and not a reduction to me, David.
You say this about refereeing at the top three journals “at this stage in the process we have a reduction of 1.7 lots of refereeing per paper.” Correct about that stage! But we’ve also released more papers upon the other journals, because we haven’t accepted so many of them. So now 90% of the initial papers remain to be accepted rather than 70%. Those are released upon the “other journals”, and have to be refereed there. You’ve merely reallocated the refereeing burden that would’ve been absorbed in the top 3 upon the other journals.
Again, no. In both scenarios, if (say) there are 1,000 papers per year, 300 are accepted by the top 3 journals, leaving 700 to be refereed by other journals. We’ve accepted exactly the same number of articles in both processes: 300. The difference is that selecting the 300 involves 2,700 refereeings in the first scenario, only 1,000 refereeings in the second scenario.
David W. Won’t this exacerbate the problem of over-specialisation? A lot of my work straddles the boundaries between the History of Philosophy, ethics and Philosophy of Logic. In the environment you seem to be suggesting it might get rejected by all the specialist journals as not quite their thing.
I concede that’s a genuine risk. I think ‘specialist’ journals should not over-police the bounds of their specialization, but it might happen.
Brian, the problem that the database is trying to solve is the lack of enough referees. Without more referees, desk-rejection becomes the norm. Lowering turnaround times would be icing on the cake, but it’s inessential, and that’s why I include the measure of speed only as an optional bonus feature.
Below is the paragraph of the post that pre-responds to your remark about “fixing the traffic problem by adding lanes to the highway; you’re just going to encourage more traffic and pretty soon the problem will be back.” Credit for refereeing would encourage less traffic on the road to authorship, because people could get credit by refereeing instead:
“Reward for refereeing would allow the drive towards professional advancement to express itself in refereeing and not just authorship. This helps to maintain the sustainability of blind review in both ways. First, it increases the supply of referees. Second, it offers those who need professional advancement an alternative to exacerbating the oversupply of papers. A public database of refereeing, consulted by university administrators, is the sort of academic infrastructure that would have these effects.”
It would be helpful if some people nervous about tenure refereed a whole lot, instead of flooding the system with papers just to scrape up an extra publication before they run out of time. That’s the choice that the database is supposed to motivate.
The problem with this Neil is that refereeing papers is a lot less fun than writing them. I got into philosophy because I wanted to write brilliant papers and to be recognised for doing so. I did not get into it because I wanted to work my way up the career ladder by writing conscientious referee’s reports, even though I have in fact written a great many conscientious referees reports. (I have refereed for 53 journals.) Refereeing is a chore at best and a torment at worst and the life of a career-referee (which you want to tempt some philosophers into to reduce the number of papers to be refereed) sounds to me like a living hell.
Charles, that’s why referees must be rewarded! Otherwise people won’t do the needed work that is less fun.
My point is Neil, that if you want to reduce the flow of papers to referee, a substantial number of young academics must be tempted to follow the refereeing career path rather than the paper-publishing career path. But refereeing is such an unpleasant business (a chore at best, a torment at worst) that no amount of career brownie points could tempt anyone to lead a life which to many of us at least would be a living hell. I’d rather be out mending the roads than be a career referee. Refereeing is part of he moral prince I pay for the privilege of being able to work on (and publish) my own stuff.
Also, Neil, I think it a bit patronising for successful publishers such as you and I to recommend a career of refereeing to others since it is a career path which I am pretty sure neither of us would have considered.
How do you know your work is brilliant if not by having them peer-reviewed? Reciprocity is ethically important. If your work was peer-reviewed, then you should give back by peer-reviewing *some* articles yourself.
Not everything about academia is about achieving personal career goals. Peer-reviewing is an epistemic function of academia. It might be a chore, but it’s necessary.
Evan:Try reading what I said before posting a sanctimonious response. I said I got into philosophy because WANTED to write brilliant papers not that my papers HAVE BEEN brilliant which I leave others to judge. I don’t need lectures on the ethics of reciprocity since with respect to refereeing I have overfulfilled my quota having refereed many times the number of papers that I have published. And though I did not make THAT boast in my original post I did say that I have written a great many conscientious referee’s reports, having refereed for 53 journals. From this you might have deduced that I do a fair bit of refereeing presumably because I think it my duty to do so. I very much doubt Evan, whoever you are, that you have done anywhere near as much refereeing as I have. Perhaps an apology would be in order?
The point is that it’s not always about what you *want*. If anything, your comment comes across as whining, which is fruitless to this discussion since we already know every job comes with some costs. This is a “no duh, what did you expect?” moment. A lot of the discussions on peer-reviewing is about trying to make it less of a hell than it is so that the cost can be bearable even though it’s necessary. In sum, I think your comment was fruitless given how reality is.
Let’s get this straight Evan. I implied and/or stated in my original post
a) that (on the whole) I don’t like refereeing
but
b) that I am a conscientious referee, and that I do quite a lot of it, having refereed for 53 journals.
The obvious inference to draw is that I do all this refereeing because I think it my duty as member of the profession. It’s part of the price I pay for the privilege of being a professional philosopher. Rather than drawing this obvious inference, you presumed to lecture me on my duties as a member of the profession, duties for which you had ample evidence (from what I had said) that I abundantly fulfil. In other words you were telling me that I had a duty to do something you had good reason to believe that I was actually doing. That’s patronising and offensive and I think I deserve an apology.
So let’s be clear. I publish about two papers per year and referee many more. I checked my records for 2021-to early 2022, during which period I published two book chapters and an unrefereed paper on my teaching. During this time refereed for the following journals, conferences and publishers (two occurrences signifying two reports):
So unless you do a great deal more refereeing than I do (how much do you do in fact?) you are not in a position to lecture me on my obligations to the profession. So how about that apology?
I actually think that refereeing papers is far more fun than turning half-written papers into finished papers!
My dream career would involve starting several exciting papers a year, and turning them over to my co-authors to finish, while reading and giving comments on lots of other people’s work (whether through the refereeing system or other informal means).
I too prefer having the initial ideas and the first flush of writing to finishing up. But with some exceptions, finishing up is a LOT more fun than refereeing papers.
I had a similar thought about how doing well in the referee rankings here might not be so desirable from an individual perspective. There’s an old joke in law schools about the competition to be on law review as like competing in a pie eating contest where the prize is more pie. If I do a pretty fair amount of refereeing – certainly so in relation to how often I send papers to journals – and try to do it quickly. (At a minimum, I’m almost always on time, even though the requested turn-around seems to be getting shorter all the time. I have one I need to finish soon that asked for 3 weeks, for example.) But, if this was just going to make me even more attractive as a referee to editors, it’s not obvious that it will be so attractive to me. I’d almost certainly have to start saying ‘no’ more often than I do now.
The prize for you isn’t just more refereeing – it’s a promotion or a better job elsewhere! Would you say no to that?
It would be good to be rewarded for it more! But there are also job advantages (much clearer ones, I think) for being on law review in law school. It seem that even if one is given more credit for refereeing, it would be like credit for “service” work generally – something, but not that much. (I should add that I’m not hostile to the idea – it might be a good or even very good one – only that I think it will have a limited benefit and might well impose more problems on people already doing a lot.
“Personally I think the root of the problem doesn’t entirely lie with the journals – it also lies with search committees and tenure committees. There are plenty of journals out there: over 250 just on DOAJ. There is no “oversupply of papers relative to space”. There is an oversupply of papers relative to the space that search committees and tenure committees have decided that they care about.”
This is exactly right; if a genie snapped their fingers and tomorrow we all woke up and understood “Tier 1 journals” consisting in 50 journals instead of “Tier 1-4 journals” as consisting in 50 journals things would improve, I think.
Any limits on “journal space” are entirely artificial. Say 10,000 philosophy papers are published each year; say each takes 1 megabytes of space; that’s 10 gigs. It’s less than one cheap thumb drive per year. On my desk is an inexpensive disk drive that would hold 2000 years of philosophy publications at that rate. “Journal space” is not a constraint here.
“Journal space” is operating here as a proxy for “reader attention”. Until we’ve solved the problems of sleep, and of work-life balance (entirely in favor of work and against life), reader attention is likely to grow at a far slower rate than disk drive space.
That’s an interesting point. I’m not sure how to cash it out empirically though. I suspect most of us ignore almost all philosophy papers. And people (like myself) who work in obscure areas would be more than happy to see quite a few more papers published in our areas. So I’m not convinced that “journal space” is a proxy for anything, but I’d be happy to be proven wrong.
May I offer a plea to extend this initiative to peer review of books. The situation has never been so dire as it is now. Not only am I as an editor receiving 10 times the number of submissions (most unsolicited by me and unaccompanied by a recommendation from an author, colleague, or series editor) as before the pandemic, but clearly my colleagues at other presses are as well, since it has never been harder to obtain peer reviews. Many projects, even from well-known or award-winning scholars, can receive up to 50 straight-out “no’s,” “too busy’s,” or no acknowledgment at all, with months passing without a single reader lined up. Even in the celebratory happenstance that two people will agree to review in a reasonable time frame, their reports are all too often months later than promised or never appear at all.
I’m absolutely NOT blaming these scholars, who are making a good-faith effort to participate in the work of their fields—it is a systemic problem—too many books for far too few places on editors’ lists and readers being authors themselves with their own work to complete in addition to all their other responsibilities, exacerbated by the additional requirements, professorial and familial, of pandemic conditions, not to mention the rarely discussed but omnipresent atmosphere of grief, sadness, stasis, even depression and despair.
Given that the book is in many cases the tenure and promotion driver, I hope that you can include it as part of your research. I would sincerely welcome suggestions from all who are involved in peer review, whether as editors, authors, or readers.
Thank you!
Wendy Lochner Publisher, Philosophy, Political Theory, Religious and Critical Life Studies Columbia University Press
I do not understand how adding another pointless numerical metric, valuing quantity over quality, to our CVs and performance evaluations is supposed to improve anything.
I also do not understand how we are supposed to create prestige for this measure by fiat. Our current measures (horrible as they are) arose in response to certain material needs (and then, as shown in the UK more than anywhere, were fetishized by rabid administrators — different problem).
Departments want to hire people who are respected as philosophers for their doing philosophy. Our current measures (publications in “top journals”, invitations to give talks, invitations for collected volumes, prizes and awars) are all proxies for this. Imperfect as they are, and fetishized to be almost useless. But that’s where they come from.
Never, ever will “oh they write a lot of reviews” be considered a proxy for this. In particular, if the proposed measure doesn’t even take quality into account (no “they write great reviews”). Sure, if someone is being asked to review for, say, Mind, this shows that they are being taken seriously by the editors — but the editors are so desperate for reviewers that it hardly counts. Also, bad reviewers aren’t excluded from the reviewer pool for the same reason. So there is no proxy for quality of one’s philosophical understanding here.
Departments have other material needs, e.g. wooing the students, attracting grant money, organization skills. Writing a lot of reviews is not a proxy for any of this.
Thus, there’s simply no reason for departments or universities to value reviewing taks (let alone *the number* of reviews). In the end, the material needs of the department will override anything we “agree upon” to value. The candidate with more promising proxies for improving the reputation of the department (and/or bringing money to the department) with “just okay” many reviews will still be chosen.
Perhaps we can go the UK route and just make everything super-duper formal, so that the procedure itself ensures our “desired” measures affect the outcome. That is, add up the number of publications, talks, prizes a candidate got (maybe weighting for the venue), add the number of reviews, add a score for their grant money successes. Highest number gets the job/tenure/promotion. But if we do this, we might as well just shut down our universities.
You say, “Departments want to hire people who are respected as philosophers for their doing philosophy.” The thing is that there are many different models of being “respected as philosophers for doing philosophy”. In recent decades, we’ve settled on “publishing papers in top journals” as the main metric by which people are respected as philosophers for doing philosophy. But in previous generations there were people like Rogers Albritton, Paul Benacerraf, and Sidney Morgenbesser who were respected as philosophers for doing philosophy, which mainly consisted in being consistently incisive interlocutors who sharpened up ideas that eventually got published in other people’s work.
Publishing papers doesn’t inherently draw students, earn grants, etc. any more than being a good reader and helpful commentator. But it’s been the easiest thing to measure, and so it has served.
who were respected as philosophers for doing philosophy, which mainly consisted in being consistently incisive interlocutors who sharpened up ideas that eventually got published in other people’s work.
I see the point, and appreciate it, but fear that _in practice_ this is often indistinguishable from the trait of being close friends with “important” and “top” people who then vouch for you as also a top philosopher, despite your lack of output. That’s not necessarily bad, but the line between it and the old-boys network starts to become pretty thin. There may be no fully happy solution here, but there was a lot to worry about with some of the older situations, especially if you’re not, for one reason or another, the type of person especially likely to do well in an old-boys network (even if some of the “old-boys” these days are women.)
I must admit that at first glance I thought Zach’s proposal was a promising fix for this problem. I trust his intentions and I appreciate that he applied himself to thinking carefully about this problem. However, after thinking more about it, I realized I have very strong objections.
Zach’s proposal aims to address both the slow turn-around in the blind review system and the failure to acknowledge refereeing as valuable academic labor by creating a database of blind-review metrics that non-philosophers can use in evaluations of tenure. This, it is alleged, may reduce pressure on the blind review system by 1) reducing the value of publications on one’s CV (which now have the status of a “golden ticket” to tenure), which may possibly discourage “flooding” journals with submissions; and 2) increasing the number of reviewers, who will have an incentive to work on refereeing. Achieving (2) seems to be the main goal of the proposal. I have three questions here. First, are there enough faculty in the position that Zach identifies as the main agents in this proposal, namely, junior faculty who will be up for tenure review at some point, to actually make a difference in the turn-around time for blind review? Last time a checked, around 73% of faculty positions are non-tenure track, and I presume many of them are submitting manuscripts, not reviewing them. Or, if they are reviewing them, they will not have an incentive to review them because they may never expect to be up for tenure. Thus, there might not even be enough people to make a difference in review turn-around, even if they did participate in the proposed metrics. There seems to be an optimization problem lurking around here, and how those problems are solved can often be unintuitive. Second, why think that peer review metrics will take hold as a real criterion in the tenure review process, I mean, as one that will actually make a difference? Those conducting the reviews may continue to think that what really matters is publishing. The presence of the metric doesn’t guarantee that others will value it in a way that makes a difference. The third question is whether Zach’s proposed “incentive” really is an incentive. To me it does not sound like much of an incentive. It sounds rather like a vague hope or promise one variety of academic labor I do may be valued as part of my promotion. As some people said, this just sounds like “gamificaton”—indeed, metrics about review turn-around seem to me like a perfect accompaniment to a productivity metric that an employer might use. I do not find the metric objectionable in itself; to the contrary, I think it is an important step toward recognizing the value of other forms of academic labor. But what I do find objectionable is that Zach’s proposal allows both the administration and the publishers to squeeze more “productivity”—in the form of more peer-reviews conducted–out of faculty without any compensation for this labor! (As Kenny said in his comment below: “in a sense, I think Neil’s proposal is to make journal refereeing a more integrated part of the job, more comparable to internal service on these committees.” Exactly my point. This outcome squeezes more unpaid productivity out of faculty.)
In this regard, I find Zach’s description of the acquisition of Publons most telling:
“Publons was acquired in 2017 by Clarivate (NYSE: CLVT), formed in 2016 when a private equity firm and an investment manager bought an intellectual property business. Valued at $17 billion, Clarivate has a “bold entrepreneurial mission to help customers reduce the time from new ideas to life-changing innovation.” Whether the mission of philosophy is to discover the truth as traditionalists claim or to change the world as Marxists claim, a bold entrepreneurial mission is unlikely to remain aligned with it. By charging wealthy institutional customers large sums to access information while denying access to philosophers who can’t afford the fees, bold entrepreneurs can profit at the expense of philosophy.”
When I think of the academic publishing industry in general, I think exactly of “bold entrepreneurs” who profit at the expense of the free labor of academics! Academics write all the articles and review each other’s work for free, while publishers erect paywalls and charge hefty subscription fees for the very same authors to access all this literature.
If I were to consider this proposal as a sort of bargain or negotiation, I would reject it. Consider the values of positive outcomes for the respective parties—publishers and acadmics/authors—relative to the uncertainties that influence those outcomes. For Publishers, the positive outcome—faster review process, more things to publish—is achieved if just *one* thing goes right: academics review articles faster. For Academics, the best-case scenario–which consists in a) relief of pressure on the blind-review system, b) reduction of peer-review turnaround time, and c) the recognition of other types of academic labor—depends on *many* things “going right,” things which are determined by their behavior as a group (i.e., optimization problem above) and by the various administrations at their universities, which they will have to convince to accept new metrics for tenure review. Relative to the uncertainties, the positive outcome is great for publishers because it relies on only *one* thing going right, i.e., academics’ willingness to perform free labor for them. But it is only *slightly* beneficial to academics, while it relies on many things going right for academics. The best-case scenario translates into more uncompensated labor with a high certainty of immediate benefit the publishers, and a low certainty of slight benefit to academics. Sometimes I find hilarious how academics so readily allow themselves to be fleeced; they will even design the system for other people to take advantage of them and call it “reasonableness.”
For me a real incentive would be some monetary compensation for dedicated work on the review process. When I worked in the private sector, one company I worked for made the meaning of incentive very clear: monetary compensation for achieving certain outcomes, not some vague promise of a more fair promotion system (notice: this is the incentive the proposal offers to get people to do more reviews!). Why don’t academics demand that publishers create some sort of compensation program for the peer review system? Academics write all the articles and review each other’s work for free(!!), while the publishers erect paywalls and charge hefty subscription fees to access all these materials. Now, I am perfectly willing to accept that publishers provide a valuable service to academia. But it does not change the fact that they profit from the free labor of academics. One might object that monetary compensation will corrupt the review process or “unblind” reviewers in some way, but I’m sure there are ways of structuring a compensation program to get around this.
I should say that I am not in principle opposed to Zach’s ideas for developing metrics for the peer review process. I think it is the right step toward recognizing peer-review as a valuable form of academic labor. I just think that his current proposal can be too easily co-opted to squeeze more unpaid “productivity” out of junior tenure-track faculty or even non-tenured faculty who may hope (in vain, I think) that the metrics will benefit them in the long-run.
1) At least if you’re tenured or tenure-track, refereeing isn’t uncompensated: it’s service work, done on your employer’s dime and your employer’s time. (If I say to my Dean ‘I spent today playing video games and watching YouTube’, she might look askance; if I tell her ‘I spent today catching up with my refereeing backlog’ she’ll be fine with it.) You can make a case that it’s a part of your job that’s under-recognized compared to other parts, but (a) most jobs, including most private-sector jobs, have some aspects like that; (b) it’s not completely unrecognised (I put my refereeing work on my annual self-report and it generally gets acknowledged); (c) the proposal we’re discussing is explicitly aimed at making it easier to recognize.
2) If you are a research academic, writing articles *definitely* isn’t uncompensated. It’s one of the primary things you are paid for and the major driver of career advancement.
3) I don’t really see how this proposal can increase aggregate time demands on academics, since it doesn’t change the aggregate amount of refereeing that gets done.
Sorry, what exactly is the problem with Publons? I’ve used it for quite some time and found it both exceptionally easy to engage with and a source of helpful data for promotion applications and such.
Neil’s OP claims: i) not enough people are using it to achieve critical mass (at least in Philosophy) and ii) it’s been bought by a private company called Clarivate which “has a “bold entrepreneurial mission to help customers reduce the time from new ideas to life-changing innovation.”
Re. i), wouldn’t it be better to encourage busy people to make more use of already-existing Publons infrastructure than encouraging people to build a new resource from the ground up? Re. ii), I don’t see why entrepreneurial aspirations constitute evidence per se that Publons is falling into poor hands. More convincing to me would be specific details about how Clarivate has (or will) changed Publons’ functionality for the worse (for the academic community).
i) Publons has been under current ownership for five years, and it hasn’t attained critical mass. I just tried to look up three philosophers I know on it. They aren’t there. You suggest building upon it, but…
ii) …according to an editor I talked to before putting this up, the sort of integration that the database requires isn’t possible with an external entity. The database requires deep and automated data-sharing between a large number of journals. For the journals to be sharing all this data (some of which may be confidential for reasons including the maintenance of blind review) they’ll have to have total confidence in the entity holding it. The only solution seems to be for the journals, collectively, to have control over that entity.
There are also the familiar concerns about letting access to important data be controlled by entities that can profit by charging for access to the data. Publons isn’t impressive enough for anyone to pay for it yet, but if it became that impressive, who knows what pricing scheme Clarivate might adopt to fulfill its bold entrepreneurial mission?
Publons does seem to have been growing though. I think I only signed up a year or two ago (though I couldn’t verify, because its profile for me has information about reviews going back to 2012!)
I do think that helping Publons grow seems like a much more promising strategy than trying to create another thing that doesn’t have a concrete strategy for how to be better while growing faster. Having cross-disciplinary integration seems really important for anyone whose work straddles disciplinary borders.
That last part is important for me. I do at least half of my refereeing for journals that are at least mostly outside of philosophy (political science, “migration studies”, law, various other random things) and if refereeing is going to be rewarded, I’d want that to be rewarded, too.
Why does Publons require deep and automated data-sharing between journals? Almost all of the reviews credited on my profile I’ve self-reported to them. They make it super-easy, you just email them your “thank you for reviewing X” letter from the journal editor, and they do the rest.
p.s. Review to Publication Ratio: 1.3:1 (Median: 0.3:1):-)
The core problem is the fact that there are 10,000 papers. The field has incentivized quantity of countable beans as an indicator of “productivity“. This is a poor index of philosophical value, in my opinion. And it also creates the refereeing problem. And many other problems as well.
Every time I see new proposals for problems like this I think to myself that a great solution was already proposed but many philosophers were too dismissive of it, so I will bring it up again and again, Jennifer Whiting’s “Slow Philosophy” proposal. Tenure would be judged on your 4 best pieces or 175 pgs of your best work. This would automatically solve the journal problem, it would get us to focus on quality over quantity and it wouldn’t punish people for having children or starting families.
https://dailynous.com/2015/12/31/a-modest-proposal-slow-philosophy-jennifer-whiting/
Exactly.
“This would automatically solve the journal problem, it would get us to focus on quality over quantity and it wouldn’t punish people for having children or starting families.”
And many others whose circumstances make it difficult to keep up with a high demand for publication!
Is it easier for people having children to focus on quality than quantity? I would think that if quality trades off against quantity, and if quantity trades off against family time, then quality would also trade off against family time, though it’s certainly not clear that it would have the *same* trade-offs.
Presumably, there is some amount of personal experience motivating the proposal, rather than abstract considerations regarding the transitivity of trade-offs. But I think it’s not too difficult to see the rationale for the proposal given that setting a cap on work considered is central to it.
Setting a cap to how much work is considered for tenure where that cap is reasonably low is an important part of the proposal. There are diminishing returns in terms of investment of time into quality. The first X hours spent refining a paper is going to pay off for the quality of that paper more than the subsequent Y hours in most cases (this is simplified, sometimes it’s a bit bumpier than that, but surely no one thinks quality increases linearly with time to infinity). This does not hold true for quantity.
If the threshold for quantity is set reasonably low in tenure considerations, then this means that philosophers who have less time to do writing due to personal circumstances can still put together a strong tenure portfolio because they still have the amount of time needed to create high quality work in sufficient quantities. There won’t be perfect parity with philosophers with more time, but it will be much closer.
This couldn’t be more true, in my opinion. I would rather have written a single, solid piece that’s helpful and enjoyable for my peers than 25 articles no one other than myself and my reviewers will ever read in this oversaturated field.
This is an interesting proposal. But I do worry about the gate-keeping. If someone is known to have reviewed a lot, then editors might just keep asking that person or somebody may just keep accepting offers.
Although it may help with career opportunities, it may end up unintentionally causing gate-keeping as the same people might end up refereeing excessively. The frequency of refereeing may not tell us much (or anything) about the quality of refereeing.
I like Neil’s proposal – I’ll say that up front. It is well reasoned and might actually work. I want to suggest an alternative idea just for consideration, which also might work – which involves a stick to Neil’s carrot. While reviewers will respond to incentives, they also respond to ego-threat. A public list that automatically updates the name of a referee when they are late for a review would be very useful. A list also updates to show when any single reviewer has not reviewed in more than three months. Everyone can see all this. Some might call this a Wall of Shame, but I prefer to think of it as a kind of Sunshine Law, used to disinfect bad reviewing. The combination of shaming reviewers for either (i) taking too long for reviews; OR (ii) not taking up enough reviews to do, would together prevent reviewers from dragging their feet or hiding under the sheets.
One worry with the stick: I often go for more than three months without being even asked to referee. I guess I just work in an obscure niche.
While I’m not necessarily opposed to Neil’s suggestion, I think there’s still a lot journals could do better without implementing this system: give referees shorter times, send more automated reminders etc. lots of journals still aren’t all that well run in this regard. Also – a fan of Whitings show philosophy proposal.
Okay, I think I am a bit clearer about my objections to Neil’s proposals. The problem as he sees it is it there is an oversupply of papers and an undersupply referees. Solution: even up the score by making refereeing more prestigious and more remunerative. Hence the need for a database and more public recognition for referees. But this is not going to even up the score unless you manage to tempt quite a lot of people into transferring their energies from writing papers to refereeing them. Being a professional referee rather than a professional researcher would have to be a viable – and indeed a reasonably popular, – career path. .But the problem with THAT is that refereeing is such a chore that it’s hard to see why anybody would want to make it their career. Honestly would anyone even THINK of getting into philosophy if all they had to look forward to, apart from teaching, was endless round of refereeing other peoples’ papers? Most of us become (or try to become) professional philosophers because we like wrestling with philosophical problems OURSELVES. I don’t mean of course that it isn’t fun to read a good paper or a good book by somebody else especially if it pertains to your interests. But what’ isn’t fun is reading books and papers which are not so good, of which there is an abundance. And most of the time that is what a professional referee would be doing. Now this is where my ad hominem to Neil comes in. Neil is a well published and creative philosopher with a solo authored book and plenty of papers to his credit. My guess is that he has ideas for papers all the time. Even if refereeing were MUCH more prestigious and MUCH more well-rewarded than it currently is, would Neil seriously consider doing a LOT less writing and a LOT more refereeing than he does at present so that refereeing became the prime focus of his non-teaching activities? I take it that the answer is ‘No’. But if Neil wouldn’t do it why would anyone else, unless perhaps that person had totally lost faith in their own abilities as a creative philosopher (in which case they might prefer leaving the profession to the drudgery of being a professional referee)?
Two more minor problems:
Thanks for your compliments regarding my work, Charles! I hope I’m holding up my end as far as refereeing goes too – while I haven’t been counting, I think I spend most years above the 15 papers/year mark that David mentions below.
Sometimes I hear that people are trying to get more publications in time for a promotion or tenure review. That’s the effort that I’d especially like to see shifted towards refereeing. Ideally, people would write only when there’s something they really care about saying. If they mainly want career advancement, their efforts should be channeled to whatever is of most value to the profession, and currently that’s refereeing.
As my reply to Cathy suggested, I’d be happy to set this all up through Publons if (1) it’s likely to achieve critical mass, (2) journal editors can automate it without security issues, and (3) Clarivate won’t paywall the database or otherwise monetize the system in a way that undermines its value. Publons is closer than anything currently existing to critical mass and automation, and I signed up for it years ago for that reason. But it’s hard for me to see how it will achieve the level of adoption and automation that the database requires to be effective, especially with uncertainty about disruptive ways in which it might be monetized if it succeeds.
I did not mean to suggest Neil that you were not doing your duty as a referee. I only meant to suggest that you would be most unlikely to make it the main focus of your non-teaching activities. Refereeing is on the whole a cost rather than a benefit of philosophy as a profession, and it is a cost that as things stand is ONLY paid by the dutiful and MAINLY by the already successful. Let’s draw a rough and ready distinction between careerist and non-careerist papers. A careerist paper is the paper you that you probably would not have written in the absence of careerist incentives. A non-careerist paper is the paper that you would probably have written even if you had a secure job with as much money and prestige as you could reasonably wish for. The distinction is indeed rough and ready and the two categories blend into one another. For instance careerist pressures have sometimes incentivised me to finish papers which I originally wrote for the love of the thing but in which I had temporarily lost interest. (For more senior philosophers there is another category: the papers you write in response to an invitation even though it would not have significantly damaged you career if you had refused the invitation. But we can forget about invitation-papers for present purpose as they are mostly written for books and encyclopaedias whether online or otherwise. )
Okay Neil. Bearing these distinctions in mind, here’s how I understand your proposal as modified and/or clarified in the course of debate. If refereeing became a more prestigious and remunerative affair, conferring more brownie points when it came to hires, promotions and tenure, then this would incentivise career-pressured philosophers to substitute a bit of refereeing for that extra careerist paper which, absent careerist pressures, they would probably prefer not to write. You are not suggesting that anybody should become a career referee. What you ARE suggesting is a system that you hope would reduce the incentives to crank out rather rubbishy papers for no other reason than to get a job, to keep a job or to get promoted, since people would have something else to substitute for that rubbishy paper on their CVs and applications. At the same time your proposal would give less dutiful philosophers more of an incentive to do their bit for the profession by doing more refereeing. Okay, that evades SOME of my objections. For what I think you have in mind is a system in which most philosophers do as much refereeing as is now done by well-cited research-successful but CONSCIENTIOUS philosophers such as yourself, namely somewhere between 15 and 20 assignments per annum. And that is clearly compatible with a research active career. So I am on board with the ends and the only question is about the means. But here I think there is a problem. To begin with your database would not be enough. You would have to somehow engineer a major cultural shift on the part of search, committees, promotion committees etc and this means a cultural shift not only on the part of philosophers but on the part of non-philosophical academics and university bureaucrats who often play a part in such decisions. Furthermore it would have to be nearly universal. Otherwise career-pressured philosophers would probably play it safe by cranking out that extra and maybe not-so-good careerist paper rather than relying their sterling records as referees to get them across the line. So I don’t think that the creation of a refereeing database (conceived as a sort of improved Publons) would be enough to shift the incentive-structures that generate the current problem, namely an oversupply of papers, many of them not very good, and an undersupply of referees. At best it would be a very tiny and not very cost-effective step in the right direction.
A few years back I did a survey of random PhilPapers users to probe various facts about peer review. One thing that surprised me is that there seems to be a lot of unused reviewing capacity. The attached graph shows that there are more respondents willing to review more as there are respondents unwilling. Unsurprisingly, the willing are a bit less senior but they might still be quite competent. I also found:
The bottom line is there seems to be underused capacity. Perhaps journal editors need to find ways to tap a greater diversity of referees?
Of course, there will still be too many submissions. Another finding from my survey is that something like 30%-40% of respondents agreed that they have recently submitted papers that, in hindsight, they agree they shouldn’t have submitted. A higher percentage of respondents report having recently had to review papers that were clearly not publishable.
This is helpful, David! It seems like the unused refereeing capacity is mostly among tenure-track junior faculty who may not be known to editors yet.
As an untenured person with a growing publication record, I’m one of those people who would not only be willing but, I think, would be fit to do more review work. Five years away from a PhD, I’m surprised I don’t get more reviewing requests. I have no idea what it takes to make oneself more openly available on that front, but not having a social media profile may be part of the problem.
As noted above the burden of refereeing falls solely on the conscientious and mainly on the already successful. The reasons are obvious. Not every philosopher is competent to referee every paper. There are vast areas of philosophy about which I simply don’t know enough to be a competent referee, and I am more wide-ranging that most. Thus the number of competent referees on any given topic is limited. But the number of referees that an editor *knows or can reasonably believe to be competent* is more limited still. Simply having published in some area is not enough. There are plenty of papers on X and Y which are not well-regarded by X and Y scholars and whose existence does not indicate that the author is someone whose judgment can be relied on. Thus editors will tend to go for philosophers with an established reputation in any given field. When a paper on X lands on their virtual desk editors look for competent referees on the basis of their X-related reputations. My guess is (and experienced editors can say me nay) that they use a judicious mixture of gossip, databases, citations counts and perhaps a little word-of-mouth about whether a given referee is a reliable bureaucratic performer. (However brilliant she or he may be as an X-scholar, it is no use sending a paper to someone who never responds to emails.) A starting point, I suspect, is often the paper’s own bibliography. I get asked to referee a lot of papers that cite me. Now you don’t have to have published *much* in a given area to be asked to referee, but what you *have* published has to be reasonably well-cited. I have only one (co-authored) paper on Truthmaker theory but I get asked to do a lot of truthmaker-related reviewing since it has been fortunate to attract a fair amount of attention. I’ve got another paper on the analytic/synthetic distinction which in my view is just as good, if not better. Sadly, if citations are anything to go by, this opinion is not widely shared: hence no referee requests.
This brings us to David’s untapped reserve of mostly untenured potential referees. Why are they untapped? Not because they would not be competent (which is true only of some them) but because they are not – and maybe *cannot* – be known to be so. Reputation and citations are the obvious proxies for refereeing expertise. It takes time to acquire a reputation or for those citation counts to pile up, and untenured faculty often have not had the time. Thus the burden of refereeing falls primarily on the already-successful. Burnout, on the part such overburdened referees is, I suggest, at least part of the explanation for the bad behaviour so often complained of. So what is to be done? The suggestion is often made that burnt out and overburdened philosophers can simply nominate some of their juniors who they think would do a decent job. Indeed most journals of include some such a suggestion in their refereeing requests (‘If you can’t do it then …’) . But this suggestion presupposes a picture of the research-successful and in-demand-referee which often fails fit the facts. The idea is that she works at a metropolitan centre, often with a big graduate school, with former students interested in her AOSs and that she is a big-time conference-goer. especially in her fields of expertise. Thus she personally knows lots of former students and/or junior colleagues who she can conscientiously recommend if the burden of refereeing becomes too much. The trouble is that you can be a research-successful philosopher and an in-demand referee without complying with this picture. Though we have some graduate students, my university does not support a large graduate school and I don’t have many former students still active in philosophy and interested in my research specialties who I can confidently recommend as substitutes for myself. Furthermore my university is a very long way from almost everywhere else and until very recently (when Zoom conferences, seminars and workshops have become more common) I have been a very intermittent conference-goer and have attended very few speciality conferences . (I am a Russell-scholar but have never physically attended a meeting the Russell society and though I work a lot on Hume I have never attended a meeting of the Hume society. I have never attended a conference in the US until I did so on Zoom a few days ago.) The upshot is that in only one of the five or six areas of expertise for which I get asked to referee (namely conspiracy theories) can I confidently recommend a reasonable number of (mostly) junior colleagues as substitutes for myself. When the burden of refereeing gets too heavy, I often have to say ‘No’ without being able to recommend anyone else. Now it may be that I am unusual amongst in-demand referees and that most of them conform to the above stereotype. But if I am not then this probably explains why that reserve of untapped potential referees is likely to remain untapped. They won’t be tapped in the first instance since editors don’t know them to be reliable and they may not be tapped in the second instance either since first-choice referees will often be unaware of their potential. What to do? Damned if I know.
It seems like a key part of your claim is that measuring will automatically lead to administrative/institutional/disciplinary valuing. But my institution measures lots of service work that I do pretty thoroughly, and values it comparatively little against my research output. My institution measures my teaching (quantity and whatever they take to be a metric of quality) and values that comparatively little against research output. The discipline at large shares this rough conception of value: Research output gets the lion’s share, everything else is comparatively minuscule. I think this is an atrocious way to value things (it is vastly out of alignment with their actual value). It seems like you’ve got the cart before the horse. We need to first get people to see how valuable it is, so that they want to reward it, and then it will be natural for people to start, say, including it on their CVs in a less “oh, I guess I can smush service work on the last page there” fashion. Then, when people are tracking it themselves, because it is valuable to them to track (because they are getting signals from others that it matters whether they do it, and how well), it will be natural for them to sign on to systems or whatever that centralize the tracking and report who is doing most/best/fastest/etc. But only when there is a carrot as well as a stick?
In explaining why universities might use the database, I began: “Research universities seek to employ experts who improve their academic reputation.” So far, the research we publish has been the main thing they can measure as an indicator of who the experts are, so they care about research a lot. If we gave them a different measure of expertise to work with, it would suit their interests to use that.
As you say, other measures might be needed to propel the database to widespread adoption. But if philosophers can get things together enough to build the database, they can probably get things together enough to popularize its use.
I have already voiced my suspicion that part of the problem with bad behaviour on the part of referees is that refereeing falls disproportionately on the more successful and more conscientious members of the profession who, after a while, start to suffer from burnout. The following is set of suggestions for chore-averse philosophers anxious to minimize their refereeing requests with a view to preserving their psychological well-being):
1) Don’t write papers that anybody else is likely to cite.
2) Take particular care not to write papers that are likely to be cited by more than twenty people.
3) Confine your scholarship to a very narrow subfield that not many people are interested in.
4) Following on from 3), take particular care not to write about a wide range of topics especially in such a way that people might actually want to read your stuff. God forbid, for example, that you should emulate one eminent philosopher of my acquaintance and write about up-to-date philosophy mind, the philosophy of biology AND ancient philosophy. If you must write about a hot topic like forgiveness, take care not to write about environmental ethics or the ethics of family relations AS WELL. The more topics you write about the more papers you may be asked to referee, hence the less you write about, the better.
5) If somebody asks you to referee a paper, either don’t reply at all or don’t reply for a long time.
6) If and when you do reply, refuse the request due to pressure of work.
7) If you make the mistake of accepting an assignment, make sure you do not referee the paper promptly.
8) As well as being tardy and/or unresponsive, take particular care to do the report in a half-assed and slovenly sort of way. Make sure your report is useless and perfunctory, preferably in such a way that the editors are likely to notice.
9) If somehow you stuff up with all of the above, try letting in an obviously bad paper (especially a spectacularly bad one) or excluding an obviously good one. With any luck the other referee will disagree and this will generate extra work for the editors who will have to search for a third referee. This will probably put you in their bad (=good) books. And of course you may get REALLY lucky and get the opportunity to let through an absolute stinker which will bring the journal into disrepute, in which case, you will never be asked to review anything for that journal ever again.
If you follow these simple suggestions, or indeed if you follow just some of them, you will, perhaps miss out on a few excellent papers but you will be spared an enournmous amount of boredom, labour and spleen. There will be a huge amount of bad-to-mediocre philosophy that you won’t have to read. You may not feel better about yourself, but you will probably feel a LOT better about your fellow-philosophers and about the profession generally. True, you won’t be able to brag about all the journals you referee for, but promotions committees, RAE assessors etc, don’t really care very much about that stuff, so the damage you do to yourself from a career point of view will be pretty minimal, especially as journal editors are honour-bound not to publicize your short-comings. Being a bad-to-non-existent referee is pretty much consequence-free. If doing your fair share of refereeing is a component of justice, it’s a part of justice that definitely does not pay. Indeed if justice isn’t justice UNLESS it pays, then it isn’t even a part of justice. Thrasymachus would not hesitate.