Transitioning a Journal to Triple-Anonymous Review


What’s involved in converting a journal’s editorial practices from single- or double-anonymous review to triple-anonymous review?

[Louise Bourgeois, “Eyes”]

The question was posed by someone interested in helping to convince journals that don’t currently employ triple-anonymous review to start doing it. Under triple-anonymous review, the author’s identity is unknown to the journal editor handling their submission as well as the reviewers. Triple-anonymous review is thought to be a way to further protect the editorial process from being influenced by biases and conflicts of interest.

Philosophy journals that currently employ triple-anonymous review include The British Journal for the Philosophy of Science, ErgoErkenntnisEthicsEuropean Journal of PhilosophyFeminist Philosophy QuarterlyJournal of Applied PhilosophyJournal of Ethics and Social PhilosophyMind, NoûsPhilosophy and Phenomenological ResearchRes Philosophica, and Thought. (See Philosophy Journals: A Crowdsourced Guide for Authors).

The transition will involve making changes to one’s editorial software system, or moving to a new system, among other things, such as possibly bringing on additional staff. The person asking about this writes: “This may seem logistically daunting. I’m hoping the readership of Daily Nous may have concrete advice about how to do the transition. What software program(s) would be good to use? How much do they cost per year? Can this transition realistically be done without the journal employing a managing editor?  (Can it be done by uncompensated or undercompensated faculty who are editing the journal while working full-time faculty jobs?)”

Those currently working at journals that make use of triple-anonymous review are especially encouraged to share the details about their systems, staffing, costs, and overall impressions of the value of triple-anonymous review.

guest
19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Douglas W. Portmore
3 months ago

My impression is that it’s neither easy nor cheap. The main problem tends to be that most editorial managerial systems have been designed with the sciences in mind, and science journals don’t tend to practice triple-blind review. Consequently, these systems make it nearly impossible to implement triple-blind review. You have to deny the editors all but reviewer status and let the managerial editor be the only one with the status of editor in the system. This means that the managerial editor will have to do a lot administrative work that would normally just be done by the editors, as, for instance, all communication has to be through the managerial editor. And given the amount of work involved I don’t see how the position of managerial editor could rightly be an unpaid one. At one of the journals that I’ve been associated with, we had no choice but to use the managerial system associated with the publishing press. At another, we ended up using a free public domain software after the editor searched and searched for a solution to this problem, but I hated this new software so much that I ended up resigning from my position with that journal. My sense, then, is that most journals would be happy to be triple-blind if there was an easy and not-too-costly way of doing so. But, unfortunately, that’s often not the case.Report

Robbie Williams
Reply to  Douglas W. Portmore
3 months ago

Ergo’s editorial management software was (I believe) specially written for that journal, implements triple-anonymization, and is an absolute joy to use, from this subeditor’s perspective, anyway.Report

Elizabeth Harman
Elizabeth Harman
Reply to  Douglas W. Portmore
3 months ago

How many hours per week is the work of managing editor, if the managing editor is doing what you describe and handling all correspondence? (I’m wondering what it would cost to pay someone to do this work.)Report

Douglas W. Portmore
Reply to  Elizabeth Harman
3 months ago

I’m not sure. I believe that he’s full-time but that he does the same work for three journals.Report

Ben Bradley
Ben Bradley
Reply to  Elizabeth Harman
3 months ago

At Ergo we have six managing editors, all tenured faculty, as a way of spreading the workload around. We usually have two people “on duty” at a time. During those times I probably spend an hour a day or more on that – sometimes a few hours a day. A lot of this is checking for anonymity issues with submitted manuscripts and corresponding with authors about it. I’m just guessing but I think if you wanted one person to do this job all the time, it would require at least 20 hours a week. We get a lot of submissions though; it could be less for a specialty journal.

We use free software designed by Jonathan Weisberg. From my perspective it works great and does everything we need. It does not have the problems Doug raises above. It does require us to have someone who can make tweaks to the program when needed. One of our MEs, Al Wilson, has the skills to do this. If you didn’t have someone like that on board, you’d have to hire someone to occasionally fix or tweak something, so that would be an additional expense.Report

Jenny Saul
3 months ago

You should add Analysis to your list! It’s been triple anonymous for a while now.Report

Beth Hannon
Beth Hannon
3 months ago

The BJPS employs various work-arounds that all substantially increase the volume of work involved, including everything Douglas Portmore mentions regarding the online submission software (Manuscript Central in our case), as well as routing all communication to authors via the journal’s email account before then forwarding on to authors, and requiring both authors and referees to contact the editorial office rather than the relevant editor.

The load is substantial and, lest delays to authors are increased even further, requires daily work. I’m firmly convinced that the benefits of triple anonymising outweigh the costs, but it does cost (and should be remunerated!).

(I’ve implemented this system at a number of different journals and am happy to discuss the practicalities in more detail with anyone wanting to try it.)Report

David Wallace
David Wallace
Reply to  Beth Hannon
3 months ago

Can you say more about why you’re “firmly convinced” it’s worth it? I’ve always been a bit skeptical of the value of triple anonymizing (and, to be honest, even of double anonymizing) but of course I haven’t had the inside view.Report

Beth Hannon
Beth Hannon
Reply to  David Wallace
3 months ago

I take it given previous conversations that your concern here is about how effective anonymization really is, especially in smaller fields (though correct me if I’m wrong!). I can only point to the numerous incorrect guesses referees regularly make—maybe once a week or so—about an author’s identity. Most commonly, it’s a grad student or early career researcher mistaken for someone more established in the field. People get it wrong with such regularity that no one should trust their guesses. And once that doubt is introduced, the hope is that it should go some way towards alleviating biases. (This is an imperfect solution, of course. Triple anonymization is the worst, except for all the other methods…)

And as Junior Faculty mentions below, any journal with a high rate of desk rejection, like the BJPS, is better off not knowing the author’s identity. It’s perhaps less about biases against certain people and more about biases in some people’s favour. There are so many borderline cases about which editors need to make a call. Would the incredibly high profile, senior philosopher have had their paper desk rejected if their identity had been known? Even if the editors predict the paper will ultimately be rejected, I think it’s more likely that the paper would have been sent out for review in at least some cases like this. (‘This paper is only so-so, but it was written by Prof. Big Name so we know lots of people will download and cite it, making our journal stats look good’, etc. etc.)

And no small benefit: Triple anonymization provides protection for editors, insofar as they can’t be accused of playing favourites or punishing nemeses—accusations that were not unheard of prior to triple anonymization! (And because the editors of the BJPS never know whose papers we’ve rejected, there’s significant easing of the psychological burden that comes with knowing you’ve rejected the paper of a friend or colleague!)Report

David Wallace
David Wallace
Reply to  Beth Hannon
3 months ago

Thanks, that’s helpful. Some of those points I hadn’t considered and wouldn’t have thought of.

You’re right in part that I’m skeptical of the actual efficacy of anonymizing. I believe you that people make mistakes all the time; equally, my own guessing success rate is 100%. (Actually, ‘guessing’ usually misdescribes it: usually I know who wrote the paper because I’ve heard a conference talk about it or read a preprint.) I suppose the conclusion to draw is that I’m anomalously good at it!

Beyond that, I’m not sure how much concrete evidence there is that in practice anonymizing actually helps. (I accept the theoretical arguments, of course). I do think it has some negative consequences:

  • I’m most concerned that it puts people off using preprint archives, out of a sense that it compromises anonymous review. That slows the dissemination of scholarship.
  • Secondarily, I think there are cases where not knowing the author of a paper messes with the assessment of papers. For instance: sometimes a paper could legitimately be criticized for plagiarizing X, unless it was written by X.
  • More controversially, I think that in a technical field – especially where the editor may not be conversant in the technical details – sometimes author ID matters. Author X submits a paper on quantum field theory. Author Y says ‘no, this is technically confused’. Author X says ‘author Y is not competent here’. I’m not convinced it’s never relevant to consider X and Y’s track record when an editor decides to reject or to call in a third referee. (If X is Ed Witten, say…)
  • If I really want to be provocative (I’m not 100% sure I know what I think here) I’m not sure I mind if desk rejections are influenced by who the author is. By their nature, desk rejections are relatively quick, imperfect things (by the time you’ve done a really careful assessment, you might as well have refereed it). If author X has published 10 papers in the journal before, I think there are (defeasible) grounds to be careful about desk-rejecting their paper that wouldn’t apply to an author with no track record of publishing in the journal. (Is that fair? No. But fairness is not the only consideration for a journal of original research.)

As a broad point: All of the above points are about maximizing the scholarly quality of journal publications, not about maximizing equity. I agree that anonymizing has at least minor benefits for equity. I think it has at least minor disadvantages for overall quality of published research. And a journal needs to balance the two. If I were persuaded that the advantages for equity are actually really major, I might shrug about the costs to scholarship. (They’re not that high.) But I’m not sure I’m convinced the benefits for equity are that major, though you’ve certainly advanced some reasons for it that I need to think about.Report

JTD
JTD
Reply to  David Wallace
3 months ago

All of the above points are about maximizing the scholarly quality of journal publications, not about maximizing equity. I agree that anonymizing has at least minor benefits for equity. I think it has at least minor disadvantages for overall quality of published research.

Anecdotally (and so far all I see in this debate is anecdote and conjecture), the opposite seems true to me in terms of anonymization and quality. Sometimes I read a paper in a top journal and think “Jeez, this is a very week paper, how it did it ever get accepted at a top journal?”. Other times I read a paper in lower ranked journal and think “Wow, this is a cracking paper, one of the best I’ve read on this topic in a while, why was it not published in a better venue where more people will notice it?”. What I find is that weak papers in top journals are mostly written by people who are high-profile or otherwise have prestige bias on their side. By contrast, strong papers in lower-ranked journals are mostly written by low-profile people without prestige bias working for them.

Assuming these observations are reliable and reflect a more general trend, the best explanation of them seems to be that the various ways in which current journal practices are not properly anonymous (including, but not limited to, the lack of triple blind review) reduce the quality of published research, at least in the most prestigious venues. If anonymous review was scrapped altogether, I can only imagine that these issues would be exacerbated and even more poor quality work would make its way into top journals and more top quality work end up in less visible venues. Therefore, quality and equity actually seem to go in the same direction–a more equitable system will also increase quality in my opinion.Report

David Wallace
David Wallace
Reply to  JTD
3 months ago

This does not match my experience of the publishing ecosystem in philosophy of physics, fwiw.Report

Beth Hannon
Beth Hannon
Reply to  Beth Hannon
3 months ago

[Sorry, David, couldn’t embed a reply in the right place]

Agreed; I don’t consider guesses those situations where the ref has seen the paper presented or has commented on a draft. I’m thinking of situations where someone says that they’ve made an educated guess that the author is X, and for whatever reason that means they can’t be impartial. That’s where the misidentification I mention occurs.

On your worries:

  • Speed of dissemination: While self-archiving will always be the fastest, there’s no need for peer review to take as long as it does. (And I take it that nothing in philosophy is so time sensitive that it can’t be slightly slower than self-archiving.) Adjusting community norms on this front is probably possible and certainly desirable!
  • Plagiarism: I’m extremely relaxed about this one. Straight-up accusations of plagiarism in this sort of case are rare; more likely is just a sharp referee report asking for more rigorous citing. And the new paper should have included citations to the old paper, and also made clear what’s new in the current paper, so that’s a failure of the author. The author’s reputation isn’t going to be harmed given anonymity, and no journal wants to publish (and who wants to read?) something that’s just a repackaged older paper.
  • Expertise: When there are conflicting opinions and the editor doesn’t feel competent to make a call, track records definitely matter, but it’s the track records of the referees that can do the work we need here—e.g. ref #1 admits to not being up to date on the literature, while ref #2 has published lots on it recently, or get a third ref involved as a tie breaker. The editor should be competent enough to know who the experts are at least. And if the referees do a good enough job in their reports to the editor, the editor should have enough competence in the area to come to a good understanding of the point of contention.
  • Desk rejects: I’ve seen enough papers from e.g. ECRs that began life as borderline desk rejects that went on to become published and well cited as a result of constructive peer review that I think avoiding potential biases at this point is really important—and not just for reasons of equity. A big name whose paper that gets unfairly desk rejected isn’t a big loss to the literature—they can still deposit their paper in an online archive and people will read it. The chances that an ECR’s paper will still be read if unfairly desk rejected is considerably lower, and that’s a loss to the community.

So I just don’t see equity and good scholarship pulling apart here. Quite the opposite: anonymization means that papers have to stand on their own scholarly merits and not the reputations of their authors.Report

David Wallace
David Wallace
Reply to  Beth Hannon
3 months ago

You make good points. I’ll think about them.

(The only thing I definitely disagree about is that it’s not a problem if people don’t self-archive and just wait for the published paper to appear. Given that papers sometimes need multiple runs at journals to get accepted, I think research interactions are way slower without self-archiving. And at least in my subfield, I think that’s a significant loss. Sure, nothing in philosophy is so critical that it *can’t* wait a bit, but that doesn’t mean the wait doesn’t slow down scholarship.)Report

Nicolas Delon
Nicolas Delon
Reply to  David Wallace
3 months ago

Here’s a half baked thought. If we’re going to have anonymized review, we should strive for triple anonymizing. The rationale is: editors wield a lot of power, probably even more than referees, so we’re not really solving a problem if we introduce potential bias at the input stage (desk reject or not, invite referees) and at the output stage (decide on the basis of referee reports). The benefits of triple anonymity seem real, although I recognize the costs of implementation. So it could be that non-anonymity and triple anonymity are both better than double anonymity. There no doubt are benefits to lifting anonymity, and perhaps we need more journals like that in philosophy.Report

Junior Faculty
Junior Faculty
3 months ago

Philosophy and Public Affairs also employs triple blind review: https://onlinelibrary.wiley.com/page/journal/10884963/homepage/forauthors.html. I entirely agree that more journals should switch to this approach. Take for example a journal like the Journal of Political Philosophy, which desk rejects a high percentage of submissions. The lack of triple blind review raises the risk that editor biases, conscious and unconscious, impact what is sent out for review and every step thereafter in the review process.Report

Sean McAleer
Sean McAleer
3 months ago

The Journal of Self-Opacity has adopted quadruple-anonymous review, under which the author’s identity is unknown to the author as well as to the journal editor handling their submission and to the reviewers.Report

JTD
JTD
3 months ago

When I was in grad school I was told by a senior, high-profile advisor to not bother submitting to two top 20 philosophy journals (which I will not name, but many will know the journals I refer to) because the general editor was known to routinely desk reject submissions by grad students without a proper look. The advisor said that this was unfair and shouldn’t happen but counseled that that’s the reality so I would do best to work around it and target journals where my work had a better chance of getting a fair review.

Looking back on this several years later, I think that it is outrageous that two top journals blatantly discriminated in this way. It’s also outrageous that it was widely known, at least by well-connected people, that this was going on, and yet the editors/journals doing it never got publicly called out for it, and many people outside the loop never found out about it.

Given this recent history in the world of philosophy journals, I think that there is a strong case for triple-blind practices at all top journals. A discipline that only recently tolerated the bad editorial practices mentioned above, and that has significant ongoing issues with prestige bias, really needs to take solid steps to restore confidence that its elite venues are not run in unfair, discriminatory ways. Triple-blind review is not just an important safe guard for biases that might still be around, but also an important symbolic step to address the wrongs of the past.Report

Tomi Francis
Tomi Francis
3 months ago

I believe The Philosophical Quarterly is also triple-blind.

(“The journal processes submitted articles anonymously: referees do not know the identity of authors; authors do not know the identity of referees; the Editors do not know the identity of authors until after the final decision on publication has been made.”)Report