How Editors Can Use PhilPeople to Find Referees
At least part of the “referee crisis” in philosophy comes from the fact that many philosophers are never or only rarely asked to referee. How can editors find these relatively untapped referees?
[Roy Lichtenstein, “Haystack #3”]
Both David Bourget and David Chalmers, he said, responded enthusiastically to this idea, and he says that Bourget (who had some data confirming the “untapped referees” factor) offered the following advice:
Our ‘find a philosopher‘ feature of philpeople is a good start. You have to pick the relevant topic quite narrowly and order by number of publications in the topic, past five years. That gives you as top hits people that are active in the area and have solid publication records. If you don’t do ‘past five years’ you get more senior people, who aren’t going to be new to you or likely to be available.
Titlebaum also reports that “Both Davids also had some great ideas for extra search functionality they could add to PhilPeople that would make it even more useful for finding referees.”
If you have feedback about the above suggestion, or have related suggestions of your own, please share them.
I think that it would be good for a wider array of people to be asked to give input when refereeing, and for there to be better tools enabling this (a good tool would maybe even give people a way to indicate when they are free or busy, so that editors have a sense of who in the referee pool is a good person to approach).
But, as is my primary axe to grind on this topic, the fundamental issue here is that huge amounts of valuable intellectual labor (basically: all service work), which are essential to the operation of our discipline, are treated as incidental aspects of our jobs that should be squeezed in when we have time. Because they are systematically undervalued (in terms of prestige, professional advancement, material compensation, the amount of time we are expected to allocate to them, graduate school time spent training people on how to perform them properly, etc.), there will continue to be variations of recurring crises no matter what patches we apply to the system, until and unless we start valuing service work in accord with the value it has for enabling the system to function.
By all means, make it easier, coordinate it better, but also: to put the horse in front of the cart, we need to actually treat service work like it matters (this shouldn’t be outlandishly hard, since it does matter, quite a bit), not like it’s just getting in the way of letting us do the stuff we really care about.
All papers should be accepted by default. Later, if they notice a problem, referees can write reports to editors and ask to reject the paper. Problem solved.
What does one earn from refereeing? In particular, what does one earn from refereeing well?
What does one loses otherwise?
Part of the problem is not only that referees are hard to find, but that some/many of them do their job poorly.
So, another (quite obvious) solution: pay referees for their job, if their reports are reliable and good.
But also another part of the problem is that papers are rejected or accepted mostly by luck, not by objective standards. Opinions and criticisms change radically from one referee to another.
Where does the money come from? I think it would have to be quite generous in order to make the system work (especially because it being paid potentially makes it easier for people to turn requests down, since they’re seen as mutually-advantageous deals, not moral obligations).
Yes, where does the money come from?
(not vouching for that particular article, it just came up first when I googled “academic publishing profits”)
Assuming for-profit publishers could be persuaded to do this, and that this is likely to incentivise people to referee, what’s the likely outcome here? It looks as though journals with for-profit publishers would have a much easier time securing referees, have faster turn-around times, and thus attract more and better papers—all to the disadvantage of the non-profits.
The costs involved (not merely paying referees but the administrative labour, software, etc.) that this would require would be well beyond most, if not all, non-profits.
My point was the reverse, actually; not that we should move to more profit-seeking journals, but that profit-seeking journals are rent-seeking and extracting the money that could be redirected to help address these things.
If you have a mechanism to induce for-profit publishers into paying their referees, I’d be fascinated to hear it. Most opponents of for-profits propose moving over to non-profit academic-run journals, which – whatever its other virtues – won’t help with this problem.
I think we could recapture a lot of that lost money and redirect it more effectively if we withdrew our labor from a system where we volunteer it to enable other people to profit, then indebt our own institutions to gain access back to the output of our own labor. But maybe the most efficient way is to volunteer our labor at every stage so that for-profit publishers can ransom our research back to us?
Of course I know that there are practical problems to paying referees.
(But here is a possible solution: authors pay a submission fee. which, as I have been told, is a normal practice in many scientific fields).
But my main point remains: “An editor asks me to referee a paper. I am very busy with my paid job and its duties. Why should I do that for free?”
Of course stressing the economic aspect is mostly provocative, since other sorts of incentives could be provided. But the economic one makes the point, I think, clear. People do not work for free, and they should not be expected to do that. Academia is no exception.
Unpaid jobs lead exactly to the current situation: you do not find staff, and when you do, they work poorly. I would have been surprised otherwise.
Refereeing is paid work, at least for tenure-line faculty: it’s part of the service component of your job. The question isn’t “why should I do this for free?”,it’s “why should I prioritize this over aspects of my job that my employer more directly incentivizes?”
(of course, even to ask that question you have to take the neoliberal attitude to workplace duty that many academics say is anathema to the life of the mind.)
I don’t know why I have an apostrophe, sorry.
Ok, I think it’s gone now.
If it is already part of the paid job, then the solution is even easier: The university paying my salary should check that I do the refereeing, exactly like it checks that I teach properly and publish properly.
However, as a matter of fact, I highly doubt that my (or possibly any) university considers it part of my duties at all. (it is never valued, never mentioned, never checked, and it is a service for a different institution).
But if you like your reformulation, the question stands in your terms: “why should I prioritize this over aspects of my job that my employer more directly incentivizes?”.
(and, yes, I do think that public regulations should revolve primarily around incentives to work).
I still think that a submission fee paid by authors, compensating refereeing, might be a possible solution with some positive side effects (the amount of submitted papers would diminish, one would chose more wisely where to submit a paper, some referees would be very happy to do that, etc.)
“However, as a matter of fact, I highly doubt that my (or possibly any) university considers it part of my duties at all. (it is never valued, never mentioned, never checked, and it is a service for a different institution).”
I include my refereeing on my annual self-report and I’ve seen it specifically acknowledged in my annual review letter.
(To check your intuitions about your university’s attitude, consider the thought-experiment I put in the previous thread: compare the attitude of your chair or Dean to being told ‘I spent today playing video games and browsing YouTube’ versus ‘I spent today catching up with my refereeing backlog’.)
I have sat on tenure and promotion meetings. I agree that they would not want to hear about my YouTube habits. But the fact that they give performative/pro forma attention to my listing of my refereeing information during such meetings (literally: that a couple of moments of time are taken to acknowledge that it is officially part of my job description, but then move back to the part that will actually move any dials for decision-making) seems *much* more significant for assessing whether it is actually taken to be part of my responsibilities.
Lewis, I think departments are different animals than college committees (as I try to explain above). If the choice is do more of your own publishing vs. doing a review, the self-interest will be with the former. But if it is do a review as well as your own work or don’t ever do reviews so as to spend your time doing something else, self interest is not so clear. David is asking you to think of things beyond the department. (And P&T committees don’t always do what a department suggests.)
Well, just a data point (with two aspects) from a member of my college’s P&T committee (which is a shitload of work). Refereeing is viewed as part of service at my institution and we use it when we assess the department’s judgements about that component of the job. Doing it is a plus. Not having any matters if you don’t do significant other service. Service isn’t the main thing and it won’t get you tenure. But it could decide a close case.
Also being asked to review is some evidence of reputation. Someone who does a lot is probably asked to do it a lot. Being asked a lot is evidence that people in other places view your research positively. I could see it coming in this way as part of a research case for one’s work, which is generally where tenure is won or lost at an R1 university.
It is worthwhile for people to realize how bureaucracies work, and that different levels of review work differently. What a college committee thinks is not the same as what a department thinks. My guess is that the department makes up its own mind largely based on its own, relatively first hand, take on the candidate – reading the work, going to seminars, interacting in other ways, etc. But College committees have to read the evidence in the file, as do outside referees who provide some of the evidence relied upon. So having something to show others value your work matters more at that level.
It would also be entirely possible that a journal editor in one’s area of expertise is among the outside referees and would be in a position to mention your work. Not super likely, but not impossible either.
Long-winded way of saying that you might actually have this work make a difference to your tenure/promotion some day.
Another way to use PhilPapers, which I learned from another person in an editorial role, is to roughly categorize the submission, and then use the corresponding PhilPapers categories to look for people who have recently written about it.
Of course, one could quibble about the PhilPapers categories. And of course, this would better if more people went to classify their and others’ own works on PhilPapers.
However, this strategy might be a starting point to diversify the referee pool beyond who comes to mind to the editor or who is in the bibliography of the author.
There are a lot of very skilled data workers among us philosophers. There should be a way to take a bibliography and then, using philpapers, get a networked web of other papers grounded in that bibliography, churning out authors.
In fact: I would love an app that does something like this. It would help find referees—but also, when I’m working on a paper, it would help me find gaps in the literature! (Though that effect would potentially lead to information cascading, or ossifying certain papers.)
This looks like a great resource and I’m glad to hear many editors have had positive experiences with PhilPeople.
In general, I think it would be very helpful to get more information so we can better determine whether there is in fact a referee shortage. As I mentioned previously, Ben Bradley’s analysis of Ergo (https://dailynous.com/2022/02/18/is-peer-review-in-philosophy-broken-beyond-reasonable-repair/#comment-430821) suggests that the problem might be overblown. It makes sense to register the severity of the problem (and determine if one even exists) before we all jump in with our favorite solutions.
My experience (but I can only speak for the JPL) is that, yes, it’s sometimes hard to find referees for a paper, but that there is no reason to declare that the centre cannot hold. Sometimes you are lucky with finding referees quickly, sometimes you need to plod on for a bit longer, but that’s all. The PhilPeople initiative is extremely welcome though and I actually found a good referee for a paper using their Find a Philosopher thing yesterday.
Now the ball is in the editor’s court. No doubt some journal editors, especially those at more elite journals, will continue to rely on a very narrow pool of referees in order to keep the discipline’s gatekeeping in the hands of an elite “inner circle”. But they can no longer get away with complaining about how hard it is to find people willing to referee and use this to justify shutting their journals down for parts of the year, or putting restrictions on who can submit and how often submissions can be made. It has now been clearly demonstrated that there is a large untapped pool of qualified yet underused referees who are willing to fill the need. And, the editors now have an easy-to-use tool allowing them to quickly access these potential referees. We will have to wait and see what the uptake of this is and whether it leads to any substantive improvements in the peer-review process.
Just going to say that PhilPapers is great as a way to find referees, if one thinks about how to use it and serch it. I’ve been doing it for about 8 years both looking for book reviewers as BR editor for one journal and now as an editor for another. More ways of doing that via PhilPeople will also be good. I actually think that the existing search engine works well except that it stops at 1000 hits, and that it puts papers without dates at the front of a search by dates which makes the early part of some searches a bit of a mashup of forthcoming and never came out.
So I really appreciate the work to make the whole interconnected web of PhilXyz.org sites better.
OTOH, for some papers it is not going to make the difficulty of finding referees much easier, and you also can’t use a search engine without some vetting of the names you turn up. Mere philosophical interest is not enough for a referee to be someone on whom you can rely. Either there will have to be published work in the area that looks strong, or there will have to be a recommendation from a person who has done such work.
That means that a slight improvement to our current system might be brought about by more people taking really seriously the request for suggestions when one declines a refereeing request.
“If you don’t do ‘past five years’ you get more senior people, who aren’t going to be new to you or likely to be available.”
Seems to me that part of the ‘crisis’ is the culture we have cultivated wherein senior members of our community are regarded as ‘too busy’ or ‘too important’ to do this kind of work. Instead, we pass it off to those early in their careers who are likely hustling to get tenure, or to be competitive for tenure-track jobs. Meanwhile, those with job security, higher salaries, prestige, etc. can’t be bothered?
I don’t think we have that culture – most senior people I know spend loads of time writing referee reports (and grant reviews, tenure letters, references, etc.) I think it’s more that senior people, being more visible/obvious choices, are already being asked a lot – the whole context of this thread was to address the fact that some people say they’re overburdened and others say they’d like to do more refereeing and don’t get asked.
Fair enough. I won’t pretend to have much experience with what senior members of our community do or do not do–I was just reading into what that quote could mean.
I will say that it does seem to me to be a fair distribution of labor to have the senior folks doing much of this thankless labor, because again: (1) they are tenured, and (2) they are paid more than junior folks. It seems fair to me that we expect those with more job security, prestige, and power within the profession to do more to support the profession that made possible that power, prestige, and job security.
Maybe PhilPeople could have a feature where people indicate how many papers they’re prepared to review within a given calendar year. Then when you referee a paper, you can update your counter to reflect how much refereeing gas you still have in the tank.