“Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.”

In the following guest post, Samantha Copeland and Lavinia Marin (Delft University of Technology) put forward the case for an “open forum” model of peer review in philosophy. The post is based on the fuller discussion and argument in their “‘It takes a village to write a really good paper’: A normative framework for peer reviewing in philosophy” which was published in Metaphilosophy earlier this year.

This is the second in a series of weekly guest posts by different authors at Daily Nous this summer.

(Posts in this series will remain pinned to the top of the homepage for several days following initial publication.)

[detail of a painting by Nidhi Mariam Jacob]

An Opportunity for Reforming Peer Review
by Samantha Copeland and Lavinia Marin

We recently published an article on peer review and reforming the way that publications are produced, reviewed, and disseminated in academic philosophy. We had been discussing this issue for a while, and more and more was being written on the problem by philosophers on social media and in print. Good points were being brought up about the dominance of the English language and its effect on how we evaluated philosophical arguments, and proposals for reform included one we found particularly interesting: the “open forum”. This format advocates for a public, iterative, and community-driven peer review process. What would this look like in academic philosophy?

Pre-publication and post-publication forums have been proposed, whether run by journals or societies, often similar to arXiv (hosted by Cornell). We propose that peer review should also happen in a similarly open context. On our proposal, an archive would host not only papers, but also comments and advice from others alongside it, as part of a discussion and improvement process initiated by authors who post their work. The system would not be anonymized.

It’s a risky idea. Take, for example, the concerns expressed over the Matthew Effect in the ongoing discussion in the literature. Heesen and Bright (2021) suggest that their proposal for a prepublication peer review of the “forum” type would not necessarily prevent cadres of well-known philosophers from “running the show” and effectively preventing outsiders from joining the discussion. We think, however, that this problem is present in systems which emphasize anonymity, even when anonymity is in place, in part, with the aim of protecting those vulnerable to bullying or unfair treatment. Exclusion of outsiders by strict adherence to guidelines can happen as easily as exclusion through the obvious bias of the reviewers. One would hope that, at least, a public, non-anonymous forum would increase pressure on commenters to justify their criticisms and increase their accountability for mistakes and ignorance.

But even more important than this is that we might have a chance to make a positive change with open forum peer review by providing more opportunities for people to create and to critically reflect on philosophy. There is also potential for educational and professional benefits for those who would be given opportunities they do not currently have or get recognition for. Changing the nature of peer review in philosophy so that it takes place in a public forum built on the norms of transparency and democracy could help spread these opportunities and benefits more widely.

Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.

In the hard sciences, peer review is often seen as a check on quality and a floodgate against bad science, yet many publications still get retracted, which is just one reason some say that peer review is a scientific experiment that failed. One of the arguments for reform in philosophy has been similar, that peer review is failing to save us from reading bad philosophy.

In our recent paper, we acknowledge that not all gatekeeping is bad. We also argue that relying on anonymity alone to prevent bias is not enough—it tends, instead, to reinforce the wrong kind of gatekeeping: keeping people out instead of instructing them on how to get “in”. While reviewers may not know where an anonymized author studied or what their gender is, other markers like writing style or citation practices can still indicate who already “belongs” and who doesn’t to those reviewers looking for such indicators and who see their role as preventing bad philosophy from being rewarded with a publication.

We think it’s more valuable to have practices and institutions in which the emphasis is on helping people get in. An open forum for reviewing, wherein the review process is an explicit and public part of developing a paper and its arguments, fits better with this than current practices.

Our proposal is inspired in part by the pedagogical value of peer review. When it is at its best, peer review even now provides philosophers with professional and disciplinary training whenever editing moves into critical engagement with the argumentation or discusses the paper’s very worth. In conferences and university hallways, we seek out each other’s reflections on our philosophical ideas and for advice on how to take them further—good peer review is good philosophical practice (You can find further details about this in the full paper.) We wouldn’t want a peer review system that, as in the sciences, simply checks for methodological soundness and conformity. Indeed, good philosophy engages even its own methodology critically, so the boundaries between acceptable and unacceptable methodologies cannot and ought not to be drawn too clearly, nor should the process be so refined and minimized. But that is what we fear will happen if reforms take the route of entrenching anonymity and focusing primarily on improving the prevention of bias and bullying. Our paper presents a different view of reform than normally taken, insofar as we want to emphasize this moment of increased dissatisfaction as an opportunity to enact a positive change in our profession.

So, we imagined instead what an ideal peer review system for academic philosophy would be like.

A lot of what goes into the making of “quality” philosophical work is, obviously, skills: at writing, argumentation, reading, creativity, imagination, and other tasks involved in the production of a written text. Such skills develop only with training. Training typically means trainers, and so the development of such skills also needs someone willing to spend their time training us: skilled instructors. Further, it is in and through the very practice of these skills that new ideas can be crafted and tested and thus, philosophical progress can be made. In an ideal system, then, there would be means for the recognition and reward for the use of one’s own skills to develop those of others (such as is done in what we think of as good peer review, aimed at strengthening rather than tearing down each other’s arguments). A bonus would be a wider acknowledgment that practicing these skills through deliberation is needed to keep honing those skills, even in our most senior years of philosophical training (i.e., no one is beyond the stage of needing peer review, we think).

Inclusion is still a serious problem in academic philosophy, and we think putting the pedagogical above the gatekeeping aims of peer review offers a chance to make that better. Speaking from the experience of one of the authors, coming from Eastern Europe, it can be quite difficult to find someone to teach you those necessary skills. The most respected philosophers usually learned to write by getting a Western-style education. But what about those who cannot leave their home country easily to get an education elsewhere? Should we accept that one is condemned to never publish in philosophy because one cannot polish the words enough to compete with more stylish writers? If you come from a disadvantaged background, then often peer review feedback may be your primary chance to get an education in how to improve your professional philosophical writing. So it’s important that the helpful, instructional elements of peer review are encouraged, while the gatekeeping role is deemphasized.

Worries about bullying and bias will still exist in an open forum model. But anonymity has not helped in the past, and there is some reason to think it can make things worse (see, for instance, here or here). And there will be monitoring work to be done. But making the reviews of our peers a part of the general educational duty of an academic philosopher would result, we think, in a fairer system overall. For example, a public forum would provide evidence for teaching and supervising effectiveness that published articles cannot now show—engaging professionally with philosophers who are more junior than oneself is itself a skill that should not only be demonstrated but needs to be honed through practice and exemplified by those who have such skills.

The philosophical community has shown that it can reflect on its own weaknesses. As we remark in our article, there have been noted improvements in philosophy: in the way that philosophy conferences are held, in increased fairness and improved norms in post-talk Q&A sessions, in more thoughtful approaches to hiring, and in more self-aware citation practices, for example. So change is possible. And as the very discussion we have waded into with our paper on peer review suggests, change is desirable for many.

We hope that our article’s normative stance shifts the burden of proof onto those who would prevent such a radical change and the burden of justification onto those who would argue against rather than for such changes in philosophy’s processes of peer review. More importantly, we hope to shift the discussion toward what is important about peer review in philosophy, and how transforming our publication and peer review processes can create opportunities for positive change in academic philosophy itself.