How to Accelerate Refereeing
People have been experimenting on journal referees and have learned some useful information. First off, whatever you do, don’t ask Professor Procrastinate to referee for you. He cannot be trusted. Second, shorten deadlines. Third, offer cash rewards or publicize how long the referees took to do the work.
We randomly assign referees to four groups: a control group with a six-week deadline to submit a referee report; a group with a four-week deadline; a cash incentive group rewarded with $100 for meeting the four-week deadline; and a social incentive group in which referees were told that their turnaround times would be publicly posted. We obtain four sets of results. First, shorter deadlines reduce the time referees take to submit reports substantially. Second, cash incentives significantly improve speed, especially in the week before the deadline. Cash payments do not crowd out intrinsic motivation: after the cash treatment ends, referees who received cash incentives are no slower than those in the four-week deadline group. Third, social incentives have smaller but significant effects on review times and are especially effective among tenured professors, who are less sensitive to deadlines and cash incentives. Fourth, all the treatments have little or no effect on rates of agreement to review, quality of reports, or review times at other journals.
The article, “What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics,” appears in the Journal of Economic Perspectives, and can be accessed here. (via Lewis Powell)
I think that imposing four-week deadlines is good idea, but there is a potential cost. It may take longer to find someone willing to referee the paper. A potential referee may just have too much going on in the next four weeks to commit to such a deadline. And I have heard editors complain that it often takes a lot of time to find qualified referees who are willing to take on the task. This, I suspect, is why many journals often say, when inviting me to referee, that they expect a report within n weeks, but that they are willing to negotiate if I need more time. So maybe a presumptive but negotiable four-week deadline would be best.
“…all the treatments have little or no effect on rates of agreement to review, quality of reports, or review times at other journals.”
Specifically, a 6-week deadline vs. a 4-week deadline only increased the agree rate by 3.5% (if I’m reading Table 2 on p. 174 correctly).
Notably though, the absolute agree rates in this study (60-70%) are much higher than for a typical philosophy journal. So it’s uncertain how this result would transpose to philosophy. But it’s still promising, I think.
I am not surprised that shorter deadlines are effective. They primarily give the task less time to fall off the referee’s radar. Most people I know agree that refereeing an article does not take six weeks or two months time, rather they hope to find the much smaller amount of time it takes some time in the next two months. The problem is that we are all unlikely to have a magic day six weeks from now rather than three weeks from now, where we have no other work demanding our attention. There is no such day for most of us. So, instead, when we are motivated to do it, it is simply because the deadline is approaching.
For a while, I’ve been trying to help improve my speed at refereeing by building referee-hours into my weekly schedule the same way I build office hours into my schedule. A time set aside where I treat refereeing as the top priority, if I have any refereeing to do, and which otherwise can be used on anything else (much as office hours are a time where interacting with students takes top priority, but if no students present themselves to meet during office hours, I can spend that time on any other work I have). The system isn’t perfect, but it does help some, I think.