Reforming Refereeing (guest post by Aaron Garrett)

Reforming Refereeing (guest post by Aaron Garrett)


The following is a guest post* by Aaron Garrett, associate professor of philosophy at Boston University. Professor Garrett recently became editor of the History of Philosophy Quarterly and asked if we could open up a discussion about reforming various aspects of article refereeing. I encourage people to contribute to the discussion and share their experiences and concerns as authors, referees, and editors.

(Some previous related discussions at Daily Nous: A Closer Look at Philosophy Journal Practices, Guarding the Guardians, Getting Credit for Peer ReviewDeciding Which Papers to Referee, Reasons You Rejected a Paper, Making Journal Statistics Publicly Available.)


Ideas for Reforming Refereeing
by Aaron Garrett

I am currently the editor of the History of Philosophy Quarterly and in my few months running the journal I’ve been thinking a lot about how to make refereeing more efficient and more rewarding. I really liked Eric’s discussion of the upsides of refereeing. I would add another upside, implicit in Eric’s discussion. You learn a lot from refereeing papers, proposals, and book manuscripts (although not from every paper, book or proposal you referee).

Eric proposed an idea for rewarding referees – publishing referees’ names with a successful paper. This could be done formally or informally, asking referees if they want to reveal themselves to the author in order to be thanked in a note. My worry is that it rewards referees accepting papers when some of the most important work is done rejecting papers. If the response is “no need to worry, it’s a marginal reward” then is this the sort of reward that would have any effect?

I wonder what people think about sending the referees all the reports when the process is completed. And whether this should be anonymous or not? I’ve really found this helpful when I’ve refereed at journals that do it to get a sense of whether I’m in step with the other referees. Relatedly should the referees be apprized of the decision? Should they be sent offprints of pieces that are published? This all seems like a good idea to me – often one feels like the report goes into an abyss and one has little sense of whether or how it had an effect.

Also how do people feel about desk rejects? The advantage of the desk reject to the author is quick turnaround. The advantage to the journal is maximizing its limited and most important resource (along with good submissions) – the time spent by expert referees towards strong publications. The disadvantage is arbitrariness. Relatedly, do people feel a few comments are necessary with a desk reject?

Finally, authors are likely aware that some referees take a long time to turn in their reports (or never turn them in). A referee who goes AWOL is a disaster for the author, in particular when they are attempting to get publications ready for tenure or the job market. This is ramified when the paper is highly specialized and it is difficult to find an appropriate referee. Fortunately this is rare.

Authors may not be aware, though, that editors spend a lot of time waiting for potential referees to respond to their requests. Sometimes potential referees never respond at all and a week or two may be lost waiting to hear back before sending a request to another referee. One way of dealing with this is an automated system that asks referees to press a button in a link as opposed to responding to the editor. The hope is that this will aid potential referees in deciding whether to referee or not quicker. The disadvantage is it is impersonal and doesn’t give potential referees the honest sense of how much their contribution to the journal and to the profession is appreciated. Impersonality may be an advantage as well. Submitters may also prefer an automated and impersonal system. Another disadvantage to journals is that the systems are expensive.

I should stress that most potential referees are not like this at all! Indeed in my few brief months I have been impressed and heartened by how serious, thoughtful, quick, and non-vindictive the referees who volunteer their time to our journal are. And for the most part the reports are models of philosophical thoughtfulness and care, real windows into expert knowledge in a subject area. Reading many of them is an education. I wish there was some way to publicly praise some of the authors of these exemplary reports! There are a few I wish I could publish!

Anyway, any thoughts in general people have about ways to make refereeing beneficial for all involved are most welcome.

Hedgehog Review
Subscribe
Notify of
guest

50 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Rachel McKinnon
Rachel McKinnon
8 years ago

One problem is preserving anonymous review itself. I suspect we all know that referees often can’t control themselves and end up searching the paper’s title, thus undermining anonymity. So here’s a solution: papers under review *do not have titles.* The title doesn’t contribute to the review process anyway, so there are no cons, only pros.

Enzo Rossi
Reply to  Rachel McKinnon
8 years ago

We considered implementing this at European J. of Political Theory (which we already made triple-blind*): no titles, just key words. Apparently it’s not possible with Manuscript Central. The software is clunky and not very fit for purpose. But it’s very difficult to get a large publisher (SAGE, in this case) to change its ways.

* I don’t think this expression is ableist, because ‘blind’ is not used metaphorically.

Michael Cholbi
Michael Cholbi
Reply to  Enzo Rossi
8 years ago

All the manuscript management systems assign numbers to submitted manuscripts. I doubt it would be difficult to have these numbers function as the titles in what (prospective) reviewers see.

Also, Aaron: Open Journal Systems is a free manuscript management system. We use it at Teaching Philosophy.

Aaron Garrett
Aaron Garrett
Reply to  Michael Cholbi
8 years ago

Thanks Michael. I’ll look into this!

Trinidad
Trinidad
Reply to  Rachel McKinnon
8 years ago

In the absence of a title a referee who ‘can’t control themselves’ will just google key phrases from the paper or lines from the abstract. This is usually sufficient to reveal the author.

Jonathan Weisberg
Jonathan Weisberg
Reply to  Trinidad
8 years ago

Omitting the title still works for papers that are only listed by title on conference websites and the like. Only if the paper itself is online is the point of omitting the title defeated. And the author can take the paper down from their own website or academia.edu page while it’s under review. After all, they are already in the process of soliciting feedback by submitting it for review. And for many authors, feedback from conference audiences, friends, and other informal channels has already been incorporated by the time they submit the paper to a journal.

In fact, I think authors *should* wait to submit a paper until it’s as polished as they can make it on their own, without referee feedback. For authors who can present at conferences and get comments from friends or colleagues, that may mean incorporating their feedback, taking the paper offline, and then submitting it. Assuming they choose to submit to a journal that’s strict about anonymous review, that is. Which, of course, they don’t have to.

shane
8 years ago

Postdoc on the job market here. I personally like desk rejections (you know what I mean) as long as they (i) quick, i.e. 1 month or less, and (ii) are coupled with the policy “paper sent to reviewers = meaningful comments expected”. That two pronged policy seems sensible because the major worry for junior authors like me is that editor and referees will take their time and I’ll get nothing from it. I don’t mind you bouncing the paper quickly, because then I don’t have much time invested and I can take my piece down the line to the next place. On the other hand, if you’re telling me that I need to give you three or four months of time, then I only want to do that if you can guarantee me some kind of return on that time in the form of meaningful feedback at the least.

I also think the questions about how to incentivize referees is important. Some of the best incentives are monetary. Most journals of course don’t have the funds to pay referees cash directly. But since many academic journals are published by large publication houses (CUP, OUP, etc) why couldn’t the editor ask the publisher for credit to offer referees against timely reviews. I have something in mind like the following policy:

“Thanks for agreeing to referee MSS#20150101328. If you return your report within two weeks, OUP will offer you $50 worth of books available from their online store in exchange for your valuable help making our journal the best it could be. If you return your report within four weeks, we can offer you $25 worth of books. If you return your report within 6 weeks, we can offer you $10 worth of credit.”

I don’t know whether such a policy is possible, since it depends on the cooperation of the press, but I’d think there could be a pretty strong business case to be made for such a policy. The only cost to the press is at most marginal cost of $50 worth of their books in stock (which I would assume to be much less than $50).

Yann Benétreau-Dupin
Yann Benétreau-Dupin
8 years ago

Potential reviewers are less likely to agree to review a paper if the review isn’t kept anonymous, and so publishing reviewers’ name might only be worth doing on a voluntary basis indeed. There’s a website intended to give credit to reviewers that aims to do just that: https://publons.com/ It’s been talked about in Nature, but I don’t know how successful it is, and it’s not used in philosophy.

I very much appreciate if a journal sends the referees all the reports. As a referee, I like it for the very reasons you put forth. As an editor (I’ve been assist. ed. for an interdisciplinary journal), I found that, most often, referee reports complement each other more than they disagree with each other. For that reason, I find it beneficial to share the reports among the referees (anonymously and only if they knew in advance that their report would be shared). And I think it’s a matter of correctness to notify the referees of the decision.

I don’t see a problem with a high rate of desk rejects. Finding reviewers is very time-consuming, and so is reviewing. It’s not the referee’s job to review a paper in order to just give the author feedback, which is what the referee is asked to do if the manuscript clearly has no chance of being accepted. Regarding the risk of arbitrariness: because the goal of the review process is to ensure the quality of what is published (rather than to ensure the quality of the entire editorial process), I see arbitrariness in acceptance as a bigger problem than arbitrariness in rejection.

Tom Cochrane
8 years ago

You don’t need to reward referees with money. People are (surprisingly) well motivated by points. Look at the way that reputational scores incentivise people on sites like . It’s also the kind of thing you can put on your CV. Points could be awarded/subtracted by the editor and/or authors based on criteria like timeliness and helpfulness, and published on the journal’s website (with referee’s consent of course).

And if you really want to stick the boot in, you can demand a certain number of points before you’re allowed to submit to the journal (or even consortium of journals).

Tom Cochrane
Reply to  Tom Cochrane
8 years ago

P.S. You might need a ‘cooling-off’ period before authors can award points (say 1 month?)

shane
Reply to  Tom Cochrane
8 years ago

Isn’t this the basic idea of the publons mentioned in the fourth comment above? How exactly would this incentivize referees? I’m not entirely clear why I would value having publons, but maybe I’m being a bit thick here. It looks to me like the publons people think you should get credit for refereeing because doing so will help you get grants or tenure. (That looks pretty optimistic). If I understand you rightly though, you’re saying people would do it just for getting points associated with a username on a website, like Karma on Reddit, or on Stack Exchange?

Tom Cochrane
Reply to  shane
8 years ago

yes, pretty much. But I would think seriously about using rights to submit as a motivator.

shane
Reply to  Tom Cochrane
8 years ago

Do you mean that in order to submit a paper to journal X, you need to have already refereed for journal X? That looks like a chicken/egg problem to me since editors only invite experts to referee and the only way to demonstrate expertise is by publishing papers. I think one is going to run into a real problem making the publication system even more clubby than it already is that way.

Simon Evnine
Simon Evnine
8 years ago

I agree that referees should be notified of the journal’s final decision and shown the reports of other referees. An offprint, if the paper is accepted, would be nice, too.

When I am asked to referee a paper, I would like to be able to see the paper before deciding. I may be uncertain, from the abstract, whether I am really competent to do it, or interested in doing it. (I once accepted to review a paper purportedly on action theory which turned out mainly to be about algebra. I had to inform the editor I was not a competent referee, but time would have been saved if I had got to look at the paper first.) Some journals allow this to happen, but most do not.

Jules
Jules
8 years ago

I recently had a great experience reviewing for the new journal Ergo. The automated system helped immensely in keeping me on task. It sent a reminder a week ahead of the deadline, reminding me of the report’s due date. When I missed the deadline, it automatically sent an email granting a one-week extension. The prompt and consistent communication helped me complete the review within 5 weeks. Within 48 hours of submitting my report, the editors emailed with the journal’s final decision and the other reviewer’s comments, which I really liked. It turns out the other reviewer and I had similar praises and misgivings, and I was able to learn from the other review, so it was beneficial to me as well.

The editors corresponded via the automated system as well as personally throughout the process. They were prompt in answering emails, and very gracious in each correspondence. In the initial request, they asked that I read the entire article and asked specifically whether I knew the author. After reading it, I didn’t, and I didn’t Google to find out who the author might be. This wasn’t a hard choice for me – I’m busy and don’t have time to scour the web to find out information I shouldn’t have in the first place. I also wanted to play by the rules so that the system of peer review could work.

At the end of the process, the editors wrote to confirm whether my comments could be sent to the author(s), and thanked me for the thorough review. Being thanked several times for writing a thorough review was enough for me to feel that it was a worthwhile experience. At the end of the day, like any task, people need to feel that their time and labor are appreciated and valued. Emails of thanks or public recognition of some sort at the end of the review process (offprints, thanks in footnotes, etc.) will incentivize people to participate in a process that’s critical in shaping the direction of the discipline.

Ted Shear
Ted Shear
8 years ago

Aaron (if I may), thanks for writing this post! Your final comment that some referee reports are so good that you wish you could publish them gives me a thought: if the journal practices triple blind review practices, then why not offer such a reviewer the opportunity to write an invited reply in the event that the original paper is published?

This seems to have a number of benefits including: (1) incentivizing especially diligent reports, (2) establishing a merit based mechanism to provide individuals the chance to get invited papers, and (3) promoting an active dialogue on papers published in your journal. Perhaps there are other benefits as well.

Of course, some logistics would need to be thought through. E.g. presumably even in the case of an invited reply, there should be *some* further review process, but this could plausibly be done more quickly with a less onerous review process. Anyway, just a thought!

Schliesser, Eric
Schliesser, Eric
Reply to  Ted Shear
8 years ago

Second!

Jurgen De Wispelaree
Jurgen De Wispelaree
Reply to  Ted Shear
8 years ago

Second that as well! The Journal of Medical Ethics starting doing exactly this a few years ago. Many of the little debates (one position paper, two short commentaries) arise from referee comments. Seems like a v. handy way to capitalize on work already done (properly).

John Grey
John Grey
Reply to  Ted Shear
8 years ago

This is the first time I’ve seen this suggestion, but it seems like one of the stronger proposals that have been floated to date. Speaking as a junior scholar, this would be an amazing motivator to together a really detailed referee report.

The only problem I can think of — if indeed this is a problem — is that such a system would strongly incentivize Accept or Revise/Resubmit verdicts over Rejections.

Ben Crowe
Ben Crowe
Reply to  Ted Shear
8 years ago

I also think that this is a fantastic suggestion. I can recall a few years back when Peter Byrne was editor of Religious Studies. He occasionally did something very much along these lines. He also allowed authors of the accepted papers the opportunity to write brief replies.

M
M
Reply to  Ted Shear
8 years ago

One possible problem with this suggestion is that it would encourage “I have a cool objection”-type reviews. I had been convinced by other online discussions on this topic that coming up with cool objections is not the primary role of a referee. Rather, we should be looking out for flaws with the paper taken on the author’s own terms, so to say. (Admittedly, I am no longer sure what the primary role of a referee is.) A very obvious, overlooked objection might be one of such flaws, but having failed to see the reviewer’s interesting, reply-worthy objection will typically not be.

Chris
Chris
Reply to  Ted Shear
8 years ago

I mentioned this before in the discussion about whether philosophy articles should be shorter, but I will mention it again. I would like to know whether the kinds of objections, replies, revisions that occur in our recent reviewing practices is not something that formerly took place in the journals themselves as part of the professional discourse. In other words, I wonder whether Ted’s suggestion (a good one!) isn’t returning journals to an older style, in which the philosophy is in the pages of the journal rather than in e-mail attachments sent from referees to editorial assistants to authors.

Aaron Garrett
Aaron Garrett
Reply to  Ted Shear
8 years ago

This is a great idea. For reasons of the structure of the HPQ and the strict length limits it wouldn’t work for us. But I think it would be great if another journal would do it.

Dan Hicks
8 years ago

An important underlying issue here has to do with the relative importance of two goals for refereeing:
(1) accepting worthwhile pieces of writing (“good papers”), and
(2) rejecting worthless pieces of writing (“bad papers”).

I get the impression that many people think #2 is much more important than #1. Insofar as that’s true, we have good reason to reject a given paper as soon as someone in the review process judges it to be worthless, on whatever grounds. So desk rejections are fine, there’s no obvious reason why referees should need to compare notes or be apprised of final decisions, etc.

However, if we take #1 to be more important, then the role of the referee is not primarily to detect any possible problems with the paper, as grounds for rejection. Rather, it’s to help the author improve the paper, working with her/him and the editor until the paper is good enough to be published. The referee is not just one of several screens through which the paper must pass; instead, he/she is another active contributor to the project of preparing the paper for publication. Then it seems quite reasonable to think that referees might collaborate with each other by sharing reports, that they would be apprised of the final decision, that their names would be noted in the published version, etc. Desk rejection only seems acceptable insofar as the paper is clearly inappropriate — topic not in the journal’s scope, not a kind of paper that the journal publishes, etc.

Abraham Graber
Abraham Graber
8 years ago

I think we ought to reconsider the practice of blind refereeing. The “Online Disinhibition Effect” is pretty well established. By making reviewers anonymous, we are essentially creating this effect for referees. Were reviewers not anonymous, I suspect that the quality of referee reports would go up significantly. Furthermore, non-anonymous refereeing would allow referee reports to serve as the start of a philosophical dialogue. Peer review can serve at least two functions: gate-keeping for journals and an opportunity for the refinement of philosophical ideas. Blind refereeing tends to push the emphasis on peer review towards the first function, perhaps to the detriment of philosophical progress. While anonymizing referees is standard practice for philosophy journals, it’s not standard practice in all disciplines (e.g., The Journal of Applied Behavior Analysis, one of the two most highly respected journals for behavioral psychology, does not make reviewers anonymous).

The only downside seems to be that reviewers might be concerned about retribution for negative reviews. But TT faculty shouldn’t be reviewing work of anybody in a position to make decisions regarding their promotion; there should be little reason to be concerned about retribution, especially if a referee’s report is well thought out. The retribution worry only seems to apply to non-TT faculty. For a variety of reasons, the practice of having non-TT faculty act as reviewers seems problematic. Acting as a referee is considered “service to the profession.” Philosophers who lack job security are likely not well positioned to dedicate time to serving the profession; expecting non-TT faculty to dedicate time to service makes light of the significant financial concerns such faculty often have.

John Grey
John Grey
Reply to  Abraham Graber
8 years ago

This is an interesting idea. My main concern is that there are other ways in which an established scholar could (and, I’ve no doubt, sometimes would) retaliate against those who recommended to reject their paper.

Another question to ask here is how this change would affect the number of people who agree to act as referees in the first place. One of the things Aaron mentioned being difficult as an editor is finding referees; I expect that although removing anonymity would increase the quality of referee reports, it would also decrease the quantity of willing referees. It increases the risk of agreeing to referee without providing any corresponding benefit to the referee, at least on the face of it.

Jonathan Livengood
Jonathan Livengood
8 years ago

RE: “Do people feel a few comments are necessary with a desk reject?”

In my opinion, every rejection requires at least *some* comments explaining the rejection. On desk rejection, the reasons could be fit with the journal or interest of the piece or quality of the writing or whatever. A single sentence would suffice in most cases. But getting no comments is frustrating. Giving no comments makes an already pretty opaque system even less transparent. Moreover, it gives no guidance to the author and potentially just pushes problems onto another journal and another set of referees.

Personal story. I have a paper that I quite like. I’ve presented it at a few conferences, and I’ve sent it to several philosophers working in the relevant area for comments. All the reactions I’ve received have been positive, and some have been enthusiastic. But over the last year and a half, the paper has been rejected FIVE times with ZERO comments. I have yet to receive a single comment from a referee or journal editor with respect to this paper. So, I have literally no idea what is going wrong. The paper seemed to be a good fit for all of the journals I submitted it to. And if there are weaknesses in the paper, they are non-obvious, since something like twenty philosophers — many of them critics — have given me substantive comments in personal communication. What do I do? Well, I move to the next journal and submit again. I have no information on which to base a revision. If all five journals rejected the paper for the same or similar reasons, then they have been wasting my time and theirs by not giving reasons for the rejections. And if the reasons are all different, that would give me a better sense of how the journals are distinct and what it is that they want.

John Grey
John Grey
Reply to  Jonathan Livengood
8 years ago

Seconding this, though I would say that desk rejections typically merit at least a paragraph rather than a sentence. (There are of course going to be exceptions — papers that are obviously the wrong fit for the journal, for instance.)

Matt Lutz
Matt Lutz
Reply to  Jonathan Livengood
8 years ago

I agree with this, although I have sometimes found even rejections without comments to be helpful. It’s very easy to get excited about one’s own work in the middle of writing a paper, and a brusque rejection can be a wake-up call that the work that I recently considered to be so very brilliant looked considerably less brilliant to someone else’s eyes. There’s no way that a rejection can be something other than an indication that something in the paper needs fixing. Maybe there is a fundamental problem with the argument of the paper. Maybe the argument is perfect, but your presentation of it is clunky. Or maybe the presentation isn’t clunky, but some moronic, unwashed troglodite is failing to understand your brilliance. But that’s a flaw, too! The best philosophy papers are so clear and compelling that even moronic, unwashed troglodites can understand their brilliance.

Getting rejections has been useful at getting me to seriously consider the possibility that my paper has some errors, and I’ve been better about revising and finding errors as a result. The good news is that one need’t wait for an unceremonious rejection to go through this thought process. I’d recommend anyone who is about to submit a paper stop first, and perform a “premortem” of the paper submission. https://hbr.org/2007/09/performing-a-project-premortem I’ve been making an effort to do this recently, and have found it helpful.

But that doesn’t mean that comment-less desk rejections are ok. If the editor read the paper well enough to understand the flaws that make it clearly unsuitable for publication, taking three to five minutes to write down what those flaws are is not a substantial increase to the time spent refereeing the paper, and does a great service to the author.

Kate Norlock
Kate Norlock
8 years ago

Yes, like Abraham I’m not greatly persuaded by tenured professors who fear retributiveness, or at least, I do not find their self-concerned fears outweigh the evidence that anonymized refereeing definitely and systematically enables egregious behavior. Some interesting arguments against me were raised in the discussion of this on FP from a few weeks ago: https://feministphilosophers.wordpress.com/2015/07/07/anonymity-of-peer-review-reports-definitely-enables-egregious-behavior/

Kathleen Lowrey
Kathleen Lowrey
8 years ago

I wonder if something that might not exactly improve refereeing but at least would make it more transparent is more stats-keeping along the lines of professional sports. It could become routinized (over time) that people acquire a “placement rate” as reviewers. What percentage of papers that you have reviewed in your career end up reaching publication? Of course “early career” stats would be essentially meaningless — if you have only reviewed 3 papers, ever, there is nothing to be learned there.
But over time, a placement rate would rack up. It wouldn’t be a perfect snapshot of how good a reviewer you are (how able you are to shape potential into publishable product). Some of it would reflect your status (“well if Professor Big Deal says it’s a good paper, it must be! Let’s publish it!”) (but actually, that would just increase the effect of high placement rate reflecting high status, which people would want to emulate). Some of it would reflect your kindness, not your critical acuity.

I suggest this because the best and most generous reviews I’ve gotten have been — I’ve suspected — from successful senior scholars, suggesting to me that generosity and success are not at all mutually exclusive. I think a lot of the nasty reviews come from people who really do have this as their only outlet for power. If I’m right about where good reviews come from, having a “high placement rate” would itself become a measure of status that academics would want to pursue. They’d be incentivized to advocate for papers sent to them for review, look for ways to make them publishable, rather than glorying in the power of a sort of withering rejection which perhaps they themselves have experienced more often than they’d like.

A down side might be people becoming more cautious about accepting papers to review: “what if it’s a dog? What if it drags down my placement rate?” But that could be corrected for by having a measure of “robustness of score” (successfully placing 1 out of 3 papers would be less robust than 10 out of 30) — expressed in a way that wouldn’t give away exactly how many papers the scholar in question had reviewed but that would still incentivize doing more reviewing. It would also be “fair” to famous scholars, who often complain about how many papers they get sent for review and how thankless it is to do a lion’s share of reviewing work. It would also incentivize younger and less famous scholars to be kinder to one another — your successful publication is my successful placement rate. There are still a limited number of slots available in journals, so this might force extra work on to editors of “which backscratching review looks the most plausible?”. But right now they are surely sometimes choosing “which eyescratching review looks less plausible?”

Of course I could be wrong, famous people in the discipline could have savage placement rates and if that number got published, then everyone could be incentivized to be “savager” to imitate that fact. Either way, it would be interesting to find out.

Anon
Anon
8 years ago

I agree with Abraham. If the referees weren’t anonymous, they would probably be a lot more polite with their feedback. I hear far too many stories about the tone of the comments being downright rude.

Richard Yetter Chappell
8 years ago

I’ve really appreciated refereeing for journals (such as JESP) that (i) apprise referees of the eventual decision, and (ii) share all referee reports (anonymously) with all referees.

Desk rejections are fine. While I don’t think authors are *owed* comments, it’s always nice to be helpful, assuming it isn’t too much extra work to include a couple of sentences by way of explanation.

ck
ck
Reply to  Richard Yetter Chappell
8 years ago

I second this and think it’s quite important. Some of the journals I regularly referee for me do not send referees details about final decision as a matter of courtesy (though in some of these cases I can informally ask the editors when I bump into them at conferences, if I remember). But others do. I find that over time I strongly prefer to referee for journals that at least keep me updated on the decision. One sometimes reads a submission and finds it interesting or novel in some respect (perhaps the paper is ready to publish or perhaps it’s just ‘close’), and one admires it so much that perhaps one would even like to cite it (if the editor publishes). Journals that do not automatically keep their referees updated (including some that do not when I have asked when sending in my report) disincentivize an already disincentivized process.

Aaron Garrett
Aaron Garrett
8 years ago

Thank you everyone for these thoughtful comments and excellent suggestions. They are very much appreciated.

Steven French
Steven French
8 years ago

Beth Hannon, the Asst Editor of the British Journal for the Philosophy of Science, has a useful blogpost about being a good referee here: http://thebjps.typepad.com/my-blog/2015/03/howtoreferee3.html

She also just put up a post about anonymisation and how not to breach it.

Steven French
Steven French
8 years ago

Just to add:
As a referee, I’m not interested in points or prizes! I do it because I think its one of the things one should do as a (reasonably) decent member of the community (yes, I know, how noble …). I do think journals should acknowledge their referees via some kind of annual thanks for example and that authors should give a nod of the head in their acknowledgments (I’m astonished at how many folk don’t have the courtesy to do that!). The idea of a placement rate looks interesting but managing that across all the various journals and ensuring that it is transparent and immune to gaming might be tricky. Again, and speaking personally, I’m not that bothered about getting credit beyond the usual acknowledgments. But if others feel they need encouragement to act as referees, all I can say is, this is what it means to be part of the academic community. And’don’t complain about it til you’ve done it!’
As an author, I obviously get frustrated over rude, unhelpful or unfocused reports (I just received 3 readers reports for a proposed edited collection that had me face-palming in disbelief …). One things journals could do, if they haven’t already, is adopt the kind of rating system that Manuscript Central incorporates which allows editors to rate referees according to timeliness and usefulness (ok, its a bit crude but it does give grounds for flagging up those referees who one should avoid).
As an editor, I have to say that one of the biggest factors in making a timely decision on a paper is associated with the referees – finding them, getting reports back etc. Given that in some cases we have gone through 5,6, 7 or more names before finding someone willing to look at the paper, each of whom may take 4-5 days or more to respond, I’m keen to avoid adopting any policy that may put people off acting as a referee or generally add to the time involved. So, de-anonymising is out as far as we’re concerned – indeed, I’m surprised by how many of our referees decline even to be named in our end of year thanks. And given the timescales involved in refereeing I think desk rejection is a reasonable policy when you have a paper that clearly does not fall within the remit, or is obviously and seriously ‘undercooked’. Indeed, we have had cases where referees and Associate Editors have responded to papers by asking us why we wasted their time when the paper concerned so clearly does not meet the standards for publication. But when we do exercise this option, the author can usually expect to hear within a couple of weeks or so.

More generally, we need not just more people willing to act as referees, but more people who can do the job well – refereeing is a skill at which one obviously becomes better with practice but there are some pointers one can give, as Beth does in her blogpost mentioned in my previous comments.

David Wallace
David Wallace
8 years ago

Replying to Kate Norlock and Abraham Graber:

I don’t think retribution, exactly, is the problem with getting tenured people to waive anonymity. Sure, if I give a very negative review to a paper by Steve French, I don’t need to fear Steve’s wrath. But I know Steve reasonably well, just as I know a large fraction of all the senior people in philosophy of physics reasonably well, and so writing a negative report on his paper and having it identified as me is predictably likely to cause awkwardness in future interactions.

Now, “this will cause awkwardness in future interactions” is a pretty first world problem, so I’m not concerned about this being unfair on people. I am concerned that there’s more of a temptation to mute criticism and to allow something to be published that probably shouldn’t be, when the author is someone I’m in frequent professional contact with – and that’s true whether or not there’s a power disparity between us. And that temptation is probably there at a subconscious level even if you consciously try to rise above it. So a risk of deanonymising, even if it’s restricted to people with tenure, is that senior people’s papers get preferential treatment. That’s worth setting against the temptation to be unnecessarily nasty that’s provided by anonymity.

Kate Norlock
Kate Norlock
8 years ago

I’ve been thinking about this a lot as an editor, too, of the new FPQ as well as occasional editor of special issues and anthologies. I started to articulate the following after posting on FP about Hilda Bastian’s recent PLOS blog post, but got sidetracked by other arguments there and didn’t fully state this: I can imagine a world in which, say, 25 years from now, Philosophy is not anonymously peer-reviewed and printed in paper journals at all. Instead, I consider it possible and preferable that short publications will be primarily online, and peer-reviewed by name-identified and selected referees right out in the internet public sphere just like book reviews or online workshops. Reasons are many:
1. We are accustomed to, but don’t need to, follow the practices of the sciences, as we’re not usually or primarily peer-reviewing for repeatable experiments and well-verified data. (Some are, I realize, but not most of us.)
2. Some of the most vibrant and interesting work being done today is conducted in open exchanges on social media and blogs, not waiting around for anonymous peer review.
3. Anonymity of referees definitely enables egregious and crummy behavior (Bastian). This is important.
4. There are decreasing numbers of tenured profs and increases in grad-student publishing, adjuncts publishing, and so on. Therefore, the demand for referees has become extremely lopsided, with the result that it takes me much longer to even find not-overburdened referees than it did ten years ago.
5. The internet exists.

Without Tenure
Without Tenure
8 years ago

As a referee, if I put in a decent amount of work explaining why I think a paper ought not be accepted by a journal, but my advice is simply ignored without comment, I will be much less likely to accept a referee request from that same editor. That may not be something to be proud of, of course. But I would be surprised if I’m alone in feeling this way.

One way of partially addressing this is to have referees see each other’s reports. This may not be enough, of course. Suppose referee 1 recommends acceptance, perhaps with some minor revisions, but referee 2 recommends rejection, and gives a long and thorough list of problems with the paper. The editor decides nonetheless to accept the paper. Seeing referee 1’s report will not help referee 2 understand why her recommendation was overridden. I’m not sure what the best thing to do in that case would be. (A note from the editor explaining referee 2 why the decision was made in spite of her recommendation would be nice, but I doubt this is something the editor *owes* the referee.)

PS: I think it is of course great that editors are able to follow only one of the two referees’ recommendation. (This is especially so in places like *Mind*, who sometimes solicit four referee reports.)

Sara L. Uckelman
Sara L. Uckelman
8 years ago

Many of these issues have been discussed over at Letters from the Editors: http://fromtheeditors.wordpress.com/

anonymous
anonymous
8 years ago

I like Mike Huemer’s idea: editors should publicly shame referees who promise a report within a particular time slot and fail to deliver. (I like it sort of jokingly, but sort of seriously.) http://owl232.net/publishing.htm

Dale Miller
8 years ago

Dear Aaron,
I’m a fan of seeing other reviewers’ reports. In fact, while this might be too complicated or time-consuming, it could actually be useful to give reviewers a chance to see each others’ reports before a decision is made or the reports are shared with authors. It seems very possible that occasionally a reviewer will want to change her mind after seeing someone else’s take on the paper. Of course, you’d want all reviewers to have written reports independently before they were shared.

While we have you here, so to speak, there was a discussion a couple of months ago in which HPQ was mentioned. I’ll paste in the URL, in case you might like to add a comment there.
http://dailynous.com/2015/05/27/what-counts-as-pre-publication/

Aaron Garrett
Aaron Garrett
Reply to  Dale Miller
8 years ago

Dear Dale,

Thanks for your comment. I know it’s the policy for HPQ, but I don’t know why the HPQ adopted the policy. I assumed that there were two reasons. First, the policy makes refereeing more anonymous. You can often figure out who wrote a paper if you really want to, but this makes it a bit more difficult. A bit like locking your windows versus leaving them open, even if locking your windows does not stop someone with a brick from going through them. Hopefully a referee’s conscience kicks in before they really scour the internet. And if not, hopefully they get bored and turn back to refereeing the paper. I think it’s a good idea for this reason. Second, putting a paper up on a very public website constitutes a form of publishing — like a broadsheet. If an essay is accepted outright, which does happen, the essay is not being published by the HPQ for the first time. If it is revised, it is publishing a version for the first time but not de novo. The journal is thus using it’s resources and referees more to certify than to publish. But the purpose of the journal is to publish new articles.

Aaron Garrett
Aaron Garrett
8 years ago

Thanks everyone for so many excellent suggestions. I will share referee’s reports with the referees and inform them of my decision, as well as send them PDFs of the successfully published articles. I may also send out the papers without titles from now on (good suggestion).

Does everyone agree that the sharing of the referee’s reports between referees should be anonymous (unless the referees choose to reveal their names)? Obviously the author of the article will remain anonymous unless the article is accepted and then they will cease to be anonymous when it is published. If I remember correctly — which I may not — the APSR shares the reports and lets each referee know who the other referees were.

Lisa S
Lisa S
Reply to  Aaron Garrett
8 years ago

While sending out a paper without a title seems like a good idea, not all papers are well titled. I’ve reviewed more than one article that was both quite interesting and whose title was very misleading as to the point of the piece (ie not alerting the reader to what was interesting in it!). In these cases, someone at some point needs to suggest a retitling. I am not sure how that happens in a paper with a suppressed title.

Aaron Garrett
Aaron Garrett
Reply to  Lisa S
8 years ago

I agree that’s a real worry. It actually comes up quite often. Perhaps have the referees look at the title if and after they give a positive report.

Jennifer Frey
Jennifer Frey
8 years ago

I think that anonymity is important to those without tenure and those who are minorities in the field. I would hate to see it compromised in *any* way. What is merely “awkward” for one person can be a professional disaster for another, since a brilliant critique may make one person look brilliant, while it paints another person as “uppity,” “bitchy” or even as someone here has suggested, just plain envious of his/her supposed betters.

As for quick turn around on responses, I think a system that sends daily reminders would be ideal. I think we all know that email can be extremely overwhelming, things get buried easily, and even the best spam filters can go horribly wrong. My charitable guess is that for many late responders, it was simply a case of thinking to oneself, I’ll get to that when I have time and then forgetting all about it. This has happened to me with important emails more times than I can count.

Aaron Garrett
Aaron Garrett
Reply to  Jennifer Frey
8 years ago

The first point seems right to me. As to the second point daily reminders seems a bit much. I would just tune them out or mark them as spam. Two weeks before due, a week before due, and then due seems more reasonable to me!

Scott Forschler
Scott Forschler
8 years ago

I have been refereeing for one journal for several years, but only get an assignment once every six months or so. And I find it both fun and stimulating–sometimes it gives me new ideas, never cribbing from the article of course, but different ways to describe my own thoughts, usually in opposition to some point made therein (or a new way to look at some source the article refers to). The upshot is: I would like to do more of this. Furthermore, I almost always review my assigned article within a week; often I even do it the same day, unless the topic is unfamiliar enough that I need to think about it more or do some background research. So, do any journal editors out there need more, and in particular faster, reviewers? I’d love to lend a hand.