Transitioning a Journal to Triple-Anonymous Review
What’s involved in converting a journal’s editorial practices from single- or double-anonymous review to triple-anonymous review?
[Louise Bourgeois, “Eyes”]
Philosophy journals that currently employ triple-anonymous review include The British Journal for the Philosophy of Science, Ergo, Erkenntnis, Ethics, European Journal of Philosophy, Feminist Philosophy Quarterly, Journal of Applied Philosophy, Journal of Ethics and Social Philosophy, Mind, Noûs, Philosophy and Phenomenological Research, Res Philosophica, and Thought. (See Philosophy Journals: A Crowdsourced Guide for Authors).
The transition will involve making changes to one’s editorial software system, or moving to a new system, among other things, such as possibly bringing on additional staff. The person asking about this writes: “This may seem logistically daunting. I’m hoping the readership of Daily Nous may have concrete advice about how to do the transition. What software program(s) would be good to use? How much do they cost per year? Can this transition realistically be done without the journal employing a managing editor? (Can it be done by uncompensated or undercompensated faculty who are editing the journal while working full-time faculty jobs?)”
Those currently working at journals that make use of triple-anonymous review are especially encouraged to share the details about their systems, staffing, costs, and overall impressions of the value of triple-anonymous review.
My impression is that it’s neither easy nor cheap. The main problem tends to be that most editorial managerial systems have been designed with the sciences in mind, and science journals don’t tend to practice triple-blind review. Consequently, these systems make it nearly impossible to implement triple-blind review. You have to deny the editors all but reviewer status and let the managerial editor be the only one with the status of editor in the system. This means that the managerial editor will have to do a lot administrative work that would normally just be done by the editors, as, for instance, all communication has to be through the managerial editor. And given the amount of work involved I don’t see how the position of managerial editor could rightly be an unpaid one. At one of the journals that I’ve been associated with, we had no choice but to use the managerial system associated with the publishing press. At another, we ended up using a free public domain software after the editor searched and searched for a solution to this problem, but I hated this new software so much that I ended up resigning from my position with that journal. My sense, then, is that most journals would be happy to be triple-blind if there was an easy and not-too-costly way of doing so. But, unfortunately, that’s often not the case.
Ergo’s editorial management software was (I believe) specially written for that journal, implements triple-anonymization, and is an absolute joy to use, from this subeditor’s perspective, anyway.
How many hours per week is the work of managing editor, if the managing editor is doing what you describe and handling all correspondence? (I’m wondering what it would cost to pay someone to do this work.)
I’m not sure. I believe that he’s full-time but that he does the same work for three journals.
At Ergo we have six managing editors, all tenured faculty, as a way of spreading the workload around. We usually have two people “on duty” at a time. During those times I probably spend an hour a day or more on that – sometimes a few hours a day. A lot of this is checking for anonymity issues with submitted manuscripts and corresponding with authors about it. I’m just guessing but I think if you wanted one person to do this job all the time, it would require at least 20 hours a week. We get a lot of submissions though; it could be less for a specialty journal.
We use free software designed by Jonathan Weisberg. From my perspective it works great and does everything we need. It does not have the problems Doug raises above. It does require us to have someone who can make tweaks to the program when needed. One of our MEs, Al Wilson, has the skills to do this. If you didn’t have someone like that on board, you’d have to hire someone to occasionally fix or tweak something, so that would be an additional expense.
You should add Analysis to your list! It’s been triple anonymous for a while now.
The BJPS employs various work-arounds that all substantially increase the volume of work involved, including everything Douglas Portmore mentions regarding the online submission software (Manuscript Central in our case), as well as routing all communication to authors via the journal’s email account before then forwarding on to authors, and requiring both authors and referees to contact the editorial office rather than the relevant editor.
The load is substantial and, lest delays to authors are increased even further, requires daily work. I’m firmly convinced that the benefits of triple anonymising outweigh the costs, but it does cost (and should be remunerated!).
(I’ve implemented this system at a number of different journals and am happy to discuss the practicalities in more detail with anyone wanting to try it.)
Can you say more about why you’re “firmly convinced” it’s worth it? I’ve always been a bit skeptical of the value of triple anonymizing (and, to be honest, even of double anonymizing) but of course I haven’t had the inside view.
I take it given previous conversations that your concern here is about how effective anonymization really is, especially in smaller fields (though correct me if I’m wrong!). I can only point to the numerous incorrect guesses referees regularly make—maybe once a week or so—about an author’s identity. Most commonly, it’s a grad student or early career researcher mistaken for someone more established in the field. People get it wrong with such regularity that no one should trust their guesses. And once that doubt is introduced, the hope is that it should go some way towards alleviating biases. (This is an imperfect solution, of course. Triple anonymization is the worst, except for all the other methods…)
And as Junior Faculty mentions below, any journal with a high rate of desk rejection, like the BJPS, is better off not knowing the author’s identity. It’s perhaps less about biases against certain people and more about biases in some people’s favour. There are so many borderline cases about which editors need to make a call. Would the incredibly high profile, senior philosopher have had their paper desk rejected if their identity had been known? Even if the editors predict the paper will ultimately be rejected, I think it’s more likely that the paper would have been sent out for review in at least some cases like this. (‘This paper is only so-so, but it was written by Prof. Big Name so we know lots of people will download and cite it, making our journal stats look good’, etc. etc.)
And no small benefit: Triple anonymization provides protection for editors, insofar as they can’t be accused of playing favourites or punishing nemeses—accusations that were not unheard of prior to triple anonymization! (And because the editors of the BJPS never know whose papers we’ve rejected, there’s significant easing of the psychological burden that comes with knowing you’ve rejected the paper of a friend or colleague!)
Thanks, that’s helpful. Some of those points I hadn’t considered and wouldn’t have thought of.
You’re right in part that I’m skeptical of the actual efficacy of anonymizing. I believe you that people make mistakes all the time; equally, my own guessing success rate is 100%. (Actually, ‘guessing’ usually misdescribes it: usually I know who wrote the paper because I’ve heard a conference talk about it or read a preprint.) I suppose the conclusion to draw is that I’m anomalously good at it!
Beyond that, I’m not sure how much concrete evidence there is that in practice anonymizing actually helps. (I accept the theoretical arguments, of course). I do think it has some negative consequences:
As a broad point: All of the above points are about maximizing the scholarly quality of journal publications, not about maximizing equity. I agree that anonymizing has at least minor benefits for equity. I think it has at least minor disadvantages for overall quality of published research. And a journal needs to balance the two. If I were persuaded that the advantages for equity are actually really major, I might shrug about the costs to scholarship. (They’re not that high.) But I’m not sure I’m convinced the benefits for equity are that major, though you’ve certainly advanced some reasons for it that I need to think about.
Anecdotally (and so far all I see in this debate is anecdote and conjecture), the opposite seems true to me in terms of anonymization and quality. Sometimes I read a paper in a top journal and think “Jeez, this is a very week paper, how it did it ever get accepted at a top journal?”. Other times I read a paper in lower ranked journal and think “Wow, this is a cracking paper, one of the best I’ve read on this topic in a while, why was it not published in a better venue where more people will notice it?”. What I find is that weak papers in top journals are mostly written by people who are high-profile or otherwise have prestige bias on their side. By contrast, strong papers in lower-ranked journals are mostly written by low-profile people without prestige bias working for them.
Assuming these observations are reliable and reflect a more general trend, the best explanation of them seems to be that the various ways in which current journal practices are not properly anonymous (including, but not limited to, the lack of triple blind review) reduce the quality of published research, at least in the most prestigious venues. If anonymous review was scrapped altogether, I can only imagine that these issues would be exacerbated and even more poor quality work would make its way into top journals and more top quality work end up in less visible venues. Therefore, quality and equity actually seem to go in the same direction–a more equitable system will also increase quality in my opinion.
This does not match my experience of the publishing ecosystem in philosophy of physics, fwiw.
[Sorry, David, couldn’t embed a reply in the right place]
Agreed; I don’t consider guesses those situations where the ref has seen the paper presented or has commented on a draft. I’m thinking of situations where someone says that they’ve made an educated guess that the author is X, and for whatever reason that means they can’t be impartial. That’s where the misidentification I mention occurs.
On your worries:
So I just don’t see equity and good scholarship pulling apart here. Quite the opposite: anonymization means that papers have to stand on their own scholarly merits and not the reputations of their authors.
You make good points. I’ll think about them.
(The only thing I definitely disagree about is that it’s not a problem if people don’t self-archive and just wait for the published paper to appear. Given that papers sometimes need multiple runs at journals to get accepted, I think research interactions are way slower without self-archiving. And at least in my subfield, I think that’s a significant loss. Sure, nothing in philosophy is so critical that it *can’t* wait a bit, but that doesn’t mean the wait doesn’t slow down scholarship.)
Here’s a half baked thought. If we’re going to have anonymized review, we should strive for triple anonymizing. The rationale is: editors wield a lot of power, probably even more than referees, so we’re not really solving a problem if we introduce potential bias at the input stage (desk reject or not, invite referees) and at the output stage (decide on the basis of referee reports). The benefits of triple anonymity seem real, although I recognize the costs of implementation. So it could be that non-anonymity and triple anonymity are both better than double anonymity. There no doubt are benefits to lifting anonymity, and perhaps we need more journals like that in philosophy.
Philosophy and Public Affairs also employs triple blind review: https://onlinelibrary.wiley.com/page/journal/10884963/homepage/forauthors.html. I entirely agree that more journals should switch to this approach. Take for example a journal like the Journal of Political Philosophy, which desk rejects a high percentage of submissions. The lack of triple blind review raises the risk that editor biases, conscious and unconscious, impact what is sent out for review and every step thereafter in the review process.
The Journal of Self-Opacity has adopted quadruple-anonymous review, under which the author’s identity is unknown to the author as well as to the journal editor handling their submission and to the reviewers.
When I was in grad school I was told by a senior, high-profile advisor to not bother submitting to two top 20 philosophy journals (which I will not name, but many will know the journals I refer to) because the general editor was known to routinely desk reject submissions by grad students without a proper look. The advisor said that this was unfair and shouldn’t happen but counseled that that’s the reality so I would do best to work around it and target journals where my work had a better chance of getting a fair review.
Looking back on this several years later, I think that it is outrageous that two top journals blatantly discriminated in this way. It’s also outrageous that it was widely known, at least by well-connected people, that this was going on, and yet the editors/journals doing it never got publicly called out for it, and many people outside the loop never found out about it.
Given this recent history in the world of philosophy journals, I think that there is a strong case for triple-blind practices at all top journals. A discipline that only recently tolerated the bad editorial practices mentioned above, and that has significant ongoing issues with prestige bias, really needs to take solid steps to restore confidence that its elite venues are not run in unfair, discriminatory ways. Triple-blind review is not just an important safe guard for biases that might still be around, but also an important symbolic step to address the wrongs of the past.
I believe The Philosophical Quarterly is also triple-blind.
(“The journal processes submitted articles anonymously: referees do not know the identity of authors; authors do not know the identity of referees; the Editors do not know the identity of authors until after the final decision on publication has been made.”)