I think a lot of problems could be solved if we found a way to make refereeing count in the way publishing does eg some exchange rate, with say 5 reports equalling one published article.

First, people who do this work get rewarded. Second, people who want to do more such work aren’t penalized relative to others. Third, with this incentive we’d eliminate the back log. Fourth, this means quicker turn around times for everyone. Fifth, that would in turn help out ECRs and make the system fairer for them.

This doesn’t really solve the free riders problem above, in that bad reviewers still benefit, but they’ll benefit less relative to the good reviewers.

Won’t this incentivise quick and dirty reports, quantity over quality? Not so. Journals can still black list people who don’t make sufficient effort, removing them from the pool, so if anyone did want to do lots of refereeing they’d have an incentive to keep the standards high.

Problems: hard to say ‘I refereed these articles’ whilst retaining anonymity over the process, and service / publishing loads are set by the university not us. These are challenges but I think some smart cookies in our discipline could figure out and coordinate something. Perhaps you can get some ‘receipt’ of your report as proof, but which doesn’t give anything away.

From the website of the American Journal of Political Science (equivalent to J Phil or Phil Review):

“Articles are selected for publication in the AJPS on the basis of a double-blind review process. Authors and co-authors of submissions to the AJPS are expected to review manuscripts for the Journal. The AJPS Editor reserves the right to refuse submissions from authors who repeatedly fail to provide reviews for the Journal when invited to do so. Any such submission refusals will be made only after consultation with at least two members of the AJPS Editorial Board.”

Thanks to Elizabeth Hannon for opening up this important discussion. I do worry that penalizing reviewers in the way proposed would risk overlooking and perhaps reinforcing the increasing institutional pressures under which academics, especially junior academics, are put. I’m keen on Tom’s suggestion that reviewers be publicly acknowledged such as by publishing their names alongside the authors’. Appreciating that it’s only a start in remodeling the value structures of publishing, the Public Philosophy Journal practices this, for instance. The PPJ also regards Public Holistic Responses, or reviewer responses published alongside a formatively peer reviewed, final work, as valuable publications in their own right that reviewers may include in their CVs and to which committees may consider giving more weight in T&P reviews.

Crazy idea: publish reviews in a digital annex alongside the papers. That way you get some name recognition for reviewing, people can tell whether you were reasonable and sufficiently critical (without being a jerk), and you can show your dean when the time comes to document your contribution to the field.

I’m appalled by the suggestion that once I’ve agreed to participate in the journal publishing system, I’m no longer allowed to take a holiday (or I’ll risk sanction if I do.) Performing free labour for publishers is one thing. Signing away my rights to take a vacation is another. (And for those of us who have substantial caring responsibilities, taking the paid time off that our contracts allow is often a key part of our childcare strategy.)

Jftr: I’ve refereed 22 times in the last 12 months. (You can check my record on Publons.) While that’s probably not as many as some people, I don’t think I’m slacking here.

On Elizabeth Hannon’s suggested alternative 4: “Good reviewers could have their own papers ‘fast tracked’ through the peer review system.” Maybe not ‘fast tracking’, but are there are some ways in which they might be able to get ‘priority’ status? For example when good reviewers submit an article they get to the top of the pile for desk consideration (or a guaranteed maximum number of days it would take for a desk reply); they get priority for a final decision when the reviews come back (I recently waited over a month for a editorial decision on an article where after the reviews had come back); maybe they are more likely to have their work sent to other ‘good reviewers’?

Two suggestions on referees (slightly at right angles to the question of how to deal with unprofessional behavior from referees, but relevant to Elizabeth Hannon’s original observation about the difficulty of finding referees):

1) in the sciences, the standard way to find competent specialist referees is to ask the author(s) of the article being refereed: indeed, recommending referees is part of the process of article submission. That would seem to save a lot of time if we adopted it in philosophy: the editor is normally not an expert on a given paper’s topic; the author (almost certainly!) is. Those suggestions don’t have to be followed, of course, but at the very least asking authors to suggest 3-4 referees (who are not their students, colleagues, supervisors or lovers) would provide a substantial extra resource.

2) Anecdotally, I gather that concerns about anonymity are a significant problem for finding referees: people refuse to referee a paper when they recognize who it’s by, either because they genuinely feel it’s inappropriate or (more cynically) because it’s an excuse to say no. And particularly in small fields, very often you are going to recognize a paper’s author. I think we would do well to reorientate anonymity from a substantive to a procedural constraint: papers should be anonymized, and referees should be enjoined not intentionally to seek out who the author is, but the contingent fact that they recognize the author shouldn’t be a reason to say know. We can still address the more serious conflict-of-interest issues directly: don’t referee papers by your students, colleagues, supervisor(s) or lover(s). (In fact, triple-blind refereeing actually gets in the way of policing those conflicts).

The natural response is that both suggestions harm the equity of the refereeing process, and indeed go somewhat against the trend of recent editorial policy. I think this is seriously overblown as a concern (the sciences manage perfectly well without even double-blind refereeing, and with authors recommending referees) but for the sake of argument let’s stipulate that both (1) and (2) increase the unfairness of the system. That doesn’t suffice to reject them. Equity is an important virtue in the editorial process but it is not the only one; it shouldn’t be given infinite weight in comparison with efficiency. And given the large and increasing strain on our system, as attested to by pretty much anyone who’s acting as a journal editor these days, there is room to consider whether we should be erring a bit more on the side of efficiency – especially as a highly inefficient and capricious system is itself a major generator of inequity.

This idea is not good. If such punishment were instituted, it would be even more difficult to find referees accepting requests to referee, for there is always a risk that you may not submit the report on time, or the report “might be meagre” or whatever, and so it would be better to decline the invitation and keep the right to submit papers to that journal. There is already little incentive to referee papers. And this would add a disincentive.

I understand the frustration of Elizabeth Hannon, but punishing is almost never the most effective thing to do.

I worry about punitive measures like this. As other commentators have noted as well, I think they’re likely to have unintended consequences. Like several other posters, the biggest one I see is that people will just become even more likely to just decline invitations to review since they’ll worry about being punished for even agreeing. And one of the main things holding up reviews is the inability to find competent reviewers who’ll agree do to it! I’ve twice had journals email me to tell me that it took two or so months to even begin a review process because they’d had trouble finding referees. Instead of going for a radical and punitive solution, I think we ought to think about instituting a number of small changes that might, when put together, have a big cumulative impact. Wallace’s suggestion that submitters suggest a few referees (or at least have the option of doing so) is a good one since it would help with the problem of finding referees. Surprenant’s suggestions that submitters pay a nominal fee and that perhaps that referees get paid are also good. People do often send half-baked work out in the hopes of getting feedback or maybe getting lucky, and a fee would discourage that. Getting paid, is, I’ve found, a pretty good incentive even if the amount is small. It changes the way you think about what you’re doing. When one isn’t paid for a service it’s all too easy to see it as a favor or as some supererogatory action; this makes it very easy to justify not doing the job well to oneself. The ugly truth is volunteers often do their jobs pretty badly in a lot of fields. But when one is paid it’s much harder to make those mental moves that justify such behavior.
I’ve a few other suggestions that I think would lead to improvement: 1. Journals ought to provide clear guidelines for referees. What are the criteria you’re looking for? What questions should I ask myself as a referee while reading this? What do you as an editor need to know in making your decisions? I’ve found that these are pretty helpful for me as a referee and lead to more detailed comments and better decisions. This would do a lot to make reports more useful I think. 2. Journal editors should take the time to personally email referees with reminders. It’s a lot harder to ignore an email from a person. This does make a difference even if it might be mildly uncomfortable for the editor. 3. Why not have it as a matter of policy that if one publishes in a journal one is committed to doing a review for them in the future and doing so in a timely fashion? Journals could have authors sign this agreement at the same time they sign the copyright or other forms. As with paying someone, I think this arrangement would also do away with the “I’m doing a nice thing here so you can’t complain if I halfass it” mindset that’s responsible for so much bad behavior on the part of referees.