An Opportunity for Reforming Peer Review (guest post)


“Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.”

In the following guest post, Samantha Copeland and Lavinia Marin (Delft University of Technology) put forward the case for an “open forum” model of peer review in philosophy. The post is based on the fuller discussion and argument in their “‘It takes a village to write a really good paper’: A normative framework for peer reviewing in philosophy” which was published in Metaphilosophy earlier this year.

This is the second in a series of weekly guest posts by different authors at Daily Nous this summer.

(Posts in this series will remain pinned to the top of the homepage for several days following initial publication.)


[detail of a painting by Nidhi Mariam Jacob]

An Opportunity for Reforming Peer Review
by Samantha Copeland and Lavinia Marin

We recently published an article on peer review and reforming the way that publications are produced, reviewed, and disseminated in academic philosophy. We had been discussing this issue for a while, and more and more was being written on the problem by philosophers on social media and in print. Good points were being brought up about the dominance of the English language and its effect on how we evaluated philosophical arguments, and proposals for reform included one we found particularly interesting: the “open forum”. This format advocates for a public, iterative, and community-driven peer review process. What would this look like in academic philosophy?

Pre-publication and post-publication forums have been proposed, whether run by journals or societies, often similar to arXiv (hosted by Cornell). We propose that peer review should also happen in a similarly open context. On our proposal, an archive would host not only papers, but also comments and advice from others alongside it, as part of a discussion and improvement process initiated by authors who post their work. The system would not be anonymized.

It’s a risky idea. Take, for example, the concerns expressed over the Matthew Effect in the ongoing discussion in the literature. Heesen and Bright (2021) suggest that their proposal for a prepublication peer review of the “forum” type would not necessarily prevent cadres of well-known philosophers from “running the show” and effectively preventing outsiders from joining the discussion. We think, however, that this problem is present in systems which emphasize anonymity, even when anonymity is in place, in part, with the aim of protecting those vulnerable to bullying or unfair treatment. Exclusion of outsiders by strict adherence to guidelines can happen as easily as exclusion through the obvious bias of the reviewers. One would hope that, at least, a public, non-anonymous forum would increase pressure on commenters to justify their criticisms and increase their accountability for mistakes and ignorance.

But even more important than this is that we might have a chance to make a positive change with open forum peer review by providing more opportunities for people to create and to critically reflect on philosophy. There is also potential for educational and professional benefits for those who would be given opportunities they do not currently have or get recognition for. Changing the nature of peer review in philosophy so that it takes place in a public forum built on the norms of transparency and democracy could help spread these opportunities and benefits more widely.

Current dissatisfaction with peer review is such an opportunity for change, so we call for taking advantage of this opportunity as fully as we can. We build our recommendations on the idea that mutual critical engagement is a skill developed through ongoing practice and actual engagement with each other’s ideas.

In the hard sciences, peer review is often seen as a check on quality and a floodgate against bad science, yet many publications still get retracted, which is just one reason some say that peer review is a scientific experiment that failed. One of the arguments for reform in philosophy has been similar, that peer review is failing to save us from reading bad philosophy.

In our recent paper, we acknowledge that not all gatekeeping is bad. We also argue that relying on anonymity alone to prevent bias is not enough—it tends, instead, to reinforce the wrong kind of gatekeeping: keeping people out instead of instructing them on how to get “in”. While reviewers may not know where an anonymized author studied or what their gender is, other markers like writing style or citation practices can still indicate who already “belongs” and who doesn’t to those reviewers looking for such indicators and who see their role as preventing bad philosophy from being rewarded with a publication.

We think it’s more valuable to have practices and institutions in which the emphasis is on helping people get in. An open forum for reviewing, wherein the review process is an explicit and public part of developing a paper and its arguments, fits better with this than current practices.

Our proposal is inspired in part by the pedagogical value of peer review. When it is at its best, peer review even now provides philosophers with professional and disciplinary training whenever editing moves into critical engagement with the argumentation or discusses the paper’s very worth. In conferences and university hallways, we seek out each other’s reflections on our philosophical ideas and for advice on how to take them further—good peer review is good philosophical practice (You can find further details about this in the full paper.) We wouldn’t want a peer review system that, as in the sciences, simply checks for methodological soundness and conformity. Indeed, good philosophy engages even its own methodology critically, so the boundaries between acceptable and unacceptable methodologies cannot and ought not to be drawn too clearly, nor should the process be so refined and minimized. But that is what we fear will happen if reforms take the route of entrenching anonymity and focusing primarily on improving the prevention of bias and bullying. Our paper presents a different view of reform than normally taken, insofar as we want to emphasize this moment of increased dissatisfaction as an opportunity to enact a positive change in our profession.

So, we imagined instead what an ideal peer review system for academic philosophy would be like.

A lot of what goes into the making of “quality” philosophical work is, obviously, skills: at writing, argumentation, reading, creativity, imagination, and other tasks involved in the production of a written text. Such skills develop only with training. Training typically means trainers, and so the development of such skills also needs someone willing to spend their time training us: skilled instructors. Further, it is in and through the very practice of these skills that new ideas can be crafted and tested and thus, philosophical progress can be made. In an ideal system, then, there would be means for the recognition and reward for the use of one’s own skills to develop those of others (such as is done in what we think of as good peer review, aimed at strengthening rather than tearing down each other’s arguments). A bonus would be a wider acknowledgment that practicing these skills through deliberation is needed to keep honing those skills, even in our most senior years of philosophical training (i.e., no one is beyond the stage of needing peer review, we think).

Inclusion is still a serious problem in academic philosophy, and we think putting the pedagogical above the gatekeeping aims of peer review offers a chance to make that better. Speaking from the experience of one of the authors, coming from Eastern Europe, it can be quite difficult to find someone to teach you those necessary skills. The most respected philosophers usually learned to write by getting a Western-style education. But what about those who cannot leave their home country easily to get an education elsewhere? Should we accept that one is condemned to never publish in philosophy because one cannot polish the words enough to compete with more stylish writers? If you come from a disadvantaged background, then often peer review feedback may be your primary chance to get an education in how to improve your professional philosophical writing. So it’s important that the helpful, instructional elements of peer review are encouraged, while the gatekeeping role is deemphasized.

Worries about bullying and bias will still exist in an open forum model. But anonymity has not helped in the past, and there is some reason to think it can make things worse (see, for instance, here or here). And there will be monitoring work to be done. But making the reviews of our peers a part of the general educational duty of an academic philosopher would result, we think, in a fairer system overall. For example, a public forum would provide evidence for teaching and supervising effectiveness that published articles cannot now show—engaging professionally with philosophers who are more junior than oneself is itself a skill that should not only be demonstrated but needs to be honed through practice and exemplified by those who have such skills.

The philosophical community has shown that it can reflect on its own weaknesses. As we remark in our article, there have been noted improvements in philosophy: in the way that philosophy conferences are held, in increased fairness and improved norms in post-talk Q&A sessions, in more thoughtful approaches to hiring, and in more self-aware citation practices, for example. So change is possible. And as the very discussion we have waded into with our paper on peer review suggests, change is desirable for many.

We hope that our article’s normative stance shifts the burden of proof onto those who would prevent such a radical change and the burden of justification onto those who would argue against rather than for such changes in philosophy’s processes of peer review. More importantly, we hope to shift the discussion toward what is important about peer review in philosophy, and how transforming our publication and peer review processes can create opportunities for positive change in academic philosophy itself.

 

Thinker Analytix: innovative tools for teaching clear and courageous thinking
Subscribe
Notify of
guest

12 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Marc Champagne
1 month ago

The standard sequence is: say something in article, then once published people may discuss it, perhaps reaching a consensus. If we start crowdsourcing reviewer feedback, we risk ending up with this less desirable sequence: say something in a draft, people must discuss it, and then the published article essentially becomes a report of the consensus that was required. I would stick to the former approach, since — despite it’s flaws — it leaves more room for dissent and less room for groupthink.

Last edited 1 month ago by Marc Champagne
Joona Räsänen
1 month ago

The problem (according to some, I did not read the linked paper so not sure if this is their view exactly) is that with the current model bad papers get published. For instance, even if I recommend a paper to be rejected, it will likely appear in another journal anyway. So the solution to the problem is to abandon peer-review and use some kind of collective post-review instead? So then everyone could upload their half-finished drafts anywhere and expect others to fix their mistakes? Then we would just have even more bad and unfinished papers to be read?

There are two problems in peer-review (of course you can disagree with me on these). First, people do not do their fair share, for example, people submit their work to a journal but refuse to review for the same journal.

Second, in philosophy, referees often miss the point of peer-review. The point is not to evaluate whether the author has managed to prove something or convince the reader. The point is to check whether the paper meets some minimum requirements regarding writing and methods and whether the author is saying something new on an important topic that people would be interested to discuss about.

Sophie sophie
Sophie sophie
1 month ago

There has already been some trials in this direction. An article published to explain this method on Colombia University Website. The journal is called Qeios. it has ‘an AI-based system finds peer reviewers suited to each submission’. Article gets published after getting three favourable reviews. ‘Reviews are then published alongside the article as they come in. Authors have the chance to respond and update their article, as needed. The reader sees the whole history of comments and changes. The entire process of vetting an article is transparent—and fast. Qeios receives a median of 10 peer reviews per article and the time from submission to publication is roughly 10 days.’ I personally this this is a much better way than the gate keepers for the journals as we have now that actively block new and alternative ideas in the field. This way we hold both the authors and the reviewers accountable.
https://www.publichealth.columbia.edu/news/new-journal-seeks-reduce-bias-scientific-publishing

Last edited 1 month ago by Sophie sophie
Charles Bakker
Charles Bakker
1 month ago

The missing part in all of this is, unfortunately, prestige. In a job market in which self-promotion is, again unfortunately, something of a prerequisite, it is unclear how replacing the old broken system would work. Perhaps one could implement a kind of “points” or “grade” system in the forum, just like other non-academic forums do? Readers can have the option of liking or upvoting strong preprints, with the reward being that the algorithm would then favour strongly liked preprints with more exposure. This could help avoid the issue of authors submitting bad manuscripts in the hopes of getting others to help them make them better. And reviewer comments could also get liked or upvoted by readers who find them to be helpful and of high quality. Not only would this tell authors which reviews are worth paying attention to, but the reviewers themselves could receive some kind of reward as well – a higher ranking, and with it, increased exposure for their own work? Finally, were this to be widely adopted, higher rankings could translate to better job prospects, if hiring committees could be convinced to pay attention to such rankings. Just spit-balling here

Mat
Mat
Reply to  Charles Bakker
1 month ago

I think this is an interesting suggestion. But it also sounds a little exhausting and not great for those of us (early career folk) who like to do our philosophy offline if we can. Starting to appreciate the status quo!

Evan
Evan
Reply to  Charles Bakker
1 month ago

These are interesting suggestions, although one potential issue is that if there is no bar to participation for readers, then the process could be driven by non-experts more than we might like.
(The quality of the average reddit comment comes to mind…).

Charles Bakker
Charles Bakker
Reply to  Evan
29 days ago

This is a fear of mine as well, but I think if it was done well, the forum would be a bit more like those dedicated forums for video games. When very specific questions are asked, such as how best to respond to Cartesian scepticism in light of X’s defence of it, for example, you will naturally attract more specific answers. I think that what would produce a Reddit-like forum would be asking questions and making claims that everyone feels more or less qualified to address.

Hugh Desmond
Hugh Desmond
29 days ago

Should the primary goal of peer-review be to educate authors?

First, education consumes a lot more mental energy than gatekeeping. That’s not to say that peer-reviewers shouldn’t strive to be constructive & courteous. But to stipulate that peer-review is primarily pedagogical means that the conscientious professional philosopher should either set aside enough time/energy to potentially educate the author on how to write, construct arguments etc. — or else not accept to do the peer-review. Wasn’t there a peer-reviewer shortage?

Second, such a policy would unleash the maximalist, overzealous peer-reviewer. Instead of checking for whether the minimum standards were met, these reviewers treat the author like a student and try to remake the submission as they themselves would have wanted to write it. Cue even more frustration. 

People have been complaining about peer-review ever since its introduction (brought to us by Xerox btw) — it’s biased, exclusionary, generally awful. But peer-review continues nevertheless because we need some type of gatekeeping is necessary to cut down on information overload. 

c b
c b
28 days ago

Pragmatically speaking, what this proposal advocates for is eliminating one of the most distinctively egalitarian features of academic philosophy: triple blinding. It remains possible in philosophy, as flawed as the system may be, for a very good paper from someone at an unranked programme to get in a very good journal purely on the basis of its quality, not the programme prestige, their advisor, their social network. And on this basis, to advance meaningfully in the profession, or at least to get your stuff read.

I have a friend who did their doctorate at an unranked programme and had been on the job market unsuccessfully for a few years, adjuncting at the institution where they did their degree. They continued to put their all into their work, and got a publication in Phil Review for a basically pathbreaking, wholly original idea. Had they published the work in a lesser-known or busier journal, it likely would have attracted little to no audience. And it certainly wouldn’t have helped them go straight from a couple years of adjuncting into a steller tenure-track job.

Admittedly, this experience might not be the norm. But this to me shows the point of the system of signalling. Allowing for the fat-tailedness of the impact of academic work, I would forthrightly argue that we ought to be more concerned with letting the best work find an audience, regardless of the class background (and thus differential in undergraduate prestige, and then graduate prestige) of the author. This kind of signalling offers at least a pathway, if only a small one. I am concerned that taking it away would rebound in favour of those with in-built audiences and in-built credibility, and it’s hard for me to find much sympathy for comparable schemes for this reason.

(I will note as a sidenote that neither author of this paper is from the States, and come from countries with comparatively minute prestige differentials across universities. The environment really isn’t like that here: having a private college coach, going to the right school, etc. is at this point necessary to get into the prestigious undergrads; the prestigious undergrads are not only favoured in grad admissions decisions but actually grade *more* generously than the less-selective state undergrads; and then the top grad programme graduates are in a position with regard to job market odds not remotely comparable to those at the bottom of the heap. It’s hard to overstate just how big the gaps at every step of the way are. In e.g. Canada or continental Europe, the prudential rationale against this scheme looks a lot different).

Tim
Tim
Reply to  c b
28 days ago

I would like to add that for those of us with imposter syndrome one positive effect of triple-blind review is that if affords some external validation that, as much as we would like to, we can’t irrationally discount as people being polite or not saying what they honestly think of our lousy work.

Last edited 28 days ago by Tim
John Alspector Finney
Reply to  c b
27 days ago

The comment by CB above is spot on and very important — and I think a lot of these people who think peer review needs completely overhauled are overlooking that the combination of blind review + very low acceptance rates (such as top five journals and the like) play an important function in the wider job market ecosystem to allow people to rise up the ranks who wouldn’t otherwise get the chance.

Rose Trappes
Rose Trappes
28 days ago

These authors make a good point that constructive, critical feedback is crucial to good philosophy and that not everyone has equal access to peer networks or training in philosophical skills. Peer review can do both things: it can provide peer feedback that improves philosophy and assist in philosophical skill development. I actually think this is true regardless of whether peer review is open or closed/anonymous.

What I was surprised about is the idea that philosophy doesn’t need a quality check function. The authors focus on bad writing, and I agree that philosophers tend to place way too much emphasis on grammar and style in their reviews. Philosophers also have the bad habit of equating bad philosophy with just whatever is not their preferred view (as the authors also point out). But there is more to bad (or good) philosophy than that.

There are things like absent, fallacious, or weak argumentation. Or not citing or engaging with literature. Or strawmanning positions in the literature. Or even plain factual errors. Not to mention philosophy papers that actually do use empirical methods (qualitative methods, digital humanities, or experiments), where all sorts of things can go astray. These are the sorts of things that peer reviewers have a duty to recognise and earmark to editors.

In their paper the authors appear to suggest that this quality check function doesn’t apply to philosophy because of methodological pluralism. But no consensus over what counts as a good method doesn’t mean there’s no way to check whether any one method has been well applied. And beyond method, there are standards of good scholarly practice that we should surely aim to see satisfied by the philosophy we are putting out into the world.

The quality check function of peer review is one reasons we want peer review, even in philosophy. Maybe current peer review isn’t performing this quality check function well. But does that mean we don’t need quality checks at all?