A Way To Increase Transparency In Academic Publishing
How can we make journal editing more transparent? That’s the question of a timely article in the recent issue of Metaphilosophy, “Why not Open the Black Box of Journal Editing in Philosophy? Make Peer Reviews of Published Papers Available,” by Caroline Schaffalitzky de Muckadell and Esben Nedenskov Petersen (both of the University of Southern Denmark).
They note that there are frequent complaints about the fairness, bias, and reliability of anonymous peer review. Here’s their suggestion:
[P]eer reviews should be made publicly available once a paper is published. This new procedure should include not only reviews from the journal accepting the paper, but also previous reviews which resulted in rejections from other journals. The idea is not to breach anonymity of the reviewers, but to put forward their arguments for public scrutiny. Accordingly, when a paper is accepted, it should be published along with anonymous reviews from the journal it has been accepted by as well as reviews from previous submissions to other journals. Previous versions of the published paper should also be made public to ensure access to the specific material assessed in the earlier reviews.
We argue that this transparency would further attempts to secure fairness and reliability of the reviewing process, or—if it is already fair and reliable— document that it is so and help put an end to misplaced suspicions. In addition, our suggestion can provide information useful for both authors, editors, reviewers, and the academic community as such.
Here are a few more reasons Schaffalitzky de Muckadell and Petersen put forward their suggestion (the numbering is added here for ease of discussion):
- “Transparency can counter reckless and abusive reviews. For instance, editors will have an interest in securing that reviews meet ethical and academic standards when they know that reviews will be made publicly available and so reflect on the journal’s reputation.”
- Seeing the multiple reviews of an article may reveal patterns of subtle bias in its treatment.
- Their suggestion can reveal patterns regarding desk rejection. “If it turned out that a journal often desk rejected papers that were subsequently published unaltered by other journals, this would call for an explanation. And similarly, if an editor tended to desk reject papers with views opposing her own this could be uncovered.”
- It will make it “easier to track correlations between, for instance, initial acceptance and later impact of revised papers. This could be useful in various discussions such as how reviews influence the quality of journal papers.”
- The practice would “provide information on the standards of academic evaluation of quality within the profession” and “may aid discussions about best practice in reviewing.”
It’s not just authors who will benefit from the increased transparency, they argue:
For journal editors the proposal offers several advantages as well. First of all, it is likely that authors will be less likely to submit work they know is unfinished just to get qualified comments on papers in progress. Second, it is a tool that can help secure quality of reviews and to sharpen a journals academic status. And third, and most importantly, the suggestion offers a transparent process to help prevent suspicion of unfairness or epistemic unreliability.
Reviewers, too, may benefit from being “able to compare (and perhaps reevaluate) their own assessment of a paper if they can see other reviews of the same paper.”
The whole paper is here (link to ungated manuscript here).
Tom Friedman, “Open Black Box”
Were this process implemented, I might take this to be a serious reason to decline reviewing anything I thought might eventually become the subject of controversy. The reviewer gets little out of reviewing; why take on uncredited burden if it runs the risk of being targeted? I worry that my thought might be rather general, and I worry then that the people willing to review such pieces would disproportionately be people at the extremes.
But I sympathize with the thoughts here. Are there ‘site review’ programs for journals as there are for departments?
It looks like the authors address a point similar to mine:
—
This concern may also be shared by prospective reviewers who will hesitate to undertake a more time-consuming task [because reviews that might be later released would call for better reviews which would take more time]. Whether increased time consumption will be justified, is something that must the object of later empirical evaluation. It is possible that open access to reviews will increase the amount of work put into writing them, but we predict that it will turn out that quality increases correspondingly to warrant it. In addition, the pressure to spend more effor[t] on reviews will be countered by the fact that reviewers remain anonymous.
In a somewhat similar vein, I’ve wondered if there could be something similar to an open access journal like PLoS One in philosophy. To some extent, blogs just are open access philosophy journals, just very informal, but you’re never going to get tenure off of a blog post, no matter how good. But where blogs differ from something like PLoS One is that the latter still does peer review for methods. Why not have an open access philosophy journal that publishes papers after a light peer review to make sure there aren’t any obvious howlers in the paper? Why not have moderated comments sections on those papers? It seems like the top journals in philosophy act much more as gatekeepers and tenure-granters than in non-humanities fields (I can’t speak for other the other humanities).
I think Craig’s comment says it all: “Were this process implemented, I might take this to be a serious reason to decline reviewing anything I thought might eventually become the subject of controversy.” If the fact that reviews are made public will make reviewers shy away from the task of reviewing, then the current system is seriously flawed. If the standards by which an academic work has been judged cannot themselves be made subject of public scrutiny I do not see how we are justified in putting our faith in these standards.
But Craig is of course right, the suggestion put forward by Schaffalitzky and Petersen would probably have the effect of shying people away from undertaking the task of reviewing, if nothing else because reviewing would be more time-consuming. But maybe this just shows that the time has come to give the reviewer the credit she deserves for the important work she undertakes.
Great idea! I’d be interested to hear more details about how to implement it, though — individuating and tracking papers as they are revised between journal submissions sounds very tricky.
How would this proposal even be implemented in practice? Is the idea that upon acceptance of a paper, the journal administrator should ask the author for a list of all the journals to which she has previously submitted the paper and the administrator would then email those journals asking for the relevant referees’ reports? For that to work there would have to be a degree of cooperation between journals and, crucially, between publishers that currently seems unattainable. Or is it that upon acceptance, the author would be expected to provide past referees’ reports herself? That option obviously seems open to abuse.
Schaffalitzky de Muckadell and Petersen touch on practical objections at the end of their paper but don’t go into detail and don’t consider the above.
It’s a long shot, but one way I could see this working is via an expanded PhilPapers service, e.g.:
1. PhilPapers adds support for (i) private (not publicly displayed) draft archiving, (ii) version control / archive history, and (iii) duplication testing [e.g. to identify drafts that share over 50% in common, or whatever standard is decided upon for individuating papers for this purpose].
2. Journals require submission of papers via PhilPapers link (a special link could be created for sharing non-public papers) to an anonymized article.
3. Journal reports and decisions are added to the private PhilPapers record for that paper.
4. When a paper is published, its associated PhilPapers entry (including journal data) is made public (and links are added to any other drafts by the author that contain substantial overlap with the current paper).
Best combined with:
5. Journal editorial boards abandon their publishers and embrace their real role as mere curators, leaving the publishing to PhilPapers itself.
Bam — instant open access for all future philosophy papers 🙂
And societies that own journals collapse and die and are no longer able to support conferences, doctoral scholarships etc. Or indeed journal administrators …
This is one of the (many) things Publons is trying to accomplish. They allow you to upload a copy of your review, reviewers’ name can be made public (just to the author(s) or to everybody else), and, if made accessible to the public, people can comment on them.
https://publons.com/about/press