Should We Get Rid of Peer Review?


“Where philosophers of science have claimed the social structure of science works well, their arguments tend to rely on things other than peer review, and that where specific benefits have been claimed for peer review, empirical research has so far failed to bear these out. Comparing this to the downsides of peer review, most prominently the massive amount of time and resources tied up in it, we conclude that we might be better off abolishing peer review”

That’s from the introduction of a new paper by Remco Heesen (Western Australia) and Liam Kofi Bright (LSE), “Is Peer Review a Good Idea?“, in the British Journal for the Philosophy of Science.

photo by J. Weinberg

According to Heesen and Bright, the empirical evidence for the effectiveness of peer review in assessing the quality of research is “mixed at best.”

Peer review’s limited effectiveness would perhaps not be a problem if it required little time and effort from scientists [which the authors use broadly to include researchers working in the sciences, social sciences, and the humanities]. But in fact the opposite is true. Going from a manuscript to a published article involves many hours of reviewing work by the assigned peer reviewers and a significant time investment from the editor handling the submission. The editor and reviewers are all scientists themselves, so the epistemic opportunity cost of their reviewing work is significant: instead of reviewing, they could be doing more science.

Their proposal is to get rid of prepublication peer review. Instead:

Scientists themselves will decide when their work is ready for sharing. When this happens, they publish their work online on something that looks like a preprint archive (think arXiv, bioRxiv, or PhilSci-archive, although the term ‘preprint’ would not be appropriate under our proposal). Authors can subsequently publish updated versions that reply to questions and comments from other scientists, which may have been provided publicly or privately. The business of journals will be to create curated collections of previously published articles. Their process for creating these collections will involve (postpublication) peer review, insofar as they currently use prepublication peer review.

Apart from avoiding the aforementioned epistemic opportunity cost, the authors claim that eliminating prepublication peer review would have other benefits, including:

  • Scientists would be able to share their discoveries more quickly.
  • More than just the privileged few who are enmeshed in particular research networks would have early and free access to scientific findings.
  • Scientists would be freer in their choices about how to spend their time, rather than being “conscripted” into reviewing manuscripts.
  • The effects of actual and expected gender bias in peer review would be reduced.
  • There would be significant savings in library budgets.
  • When it comes to assessing researchers, the role of the “short-run credit” of publication in a high-impact journal may be reduced, while that of the “long-run credit” of citation metrics may be increased.
  • The power of gatekeepers would be reduced, making the evaluation of research a “more in line with general communal norms accepted within science.”

You can read the full paper here. Discussion welcome.

Subscribe
Notify of
guest

46 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
c
c
4 years ago

A first thought on this:
“The editor and reviewers are all scientists themselves, so the epistemic opportunity cost of their reviewing work is significant: instead of reviewing, they could be doing more science.”

This seems to assume that reviewing is not an important part of science. From the list of benefits of the removal of peer review, the authors do seem to believe that peer review is not needed for good science, in fact it’s bad for science. I can buy that peer review has downsides, but I am not so sure it is as dispensable as the authors claim.

Wish I were a PI
Wish I were a PI
Reply to  c
4 years ago

Agreed here; this passage stuck out to me and I think it misunderstands what a lot of scientists actually do. I’ve worked in and around a lot of labs in neuroscience and other areas of biology, and most PI’s do very little of the “science” themselves, if any at all. For the most part they write grants and travel around giving talks at other places, and their peer-reviewing often happens on planes and in other moments like that when they couldn’t be doing “science” anyway, whatever that means. The post-docs and the grad students are the ones “doing” the science, so freeing up the PIs from peer-review isn’t going to change that much. I’m happy to hear otherwise about this.

Philodemus
Philodemus
4 years ago

Yes!

—untenured prof whose reply piece to an Analysis paper just got rejected based on exactly one uncharitable review

Eh?
Eh?
Reply to  Philodemus
4 years ago

Philodemus… your experience sounds perfectly normal. It sounds like (and correct me if I’m wrong) one referee rejected your paper, and so Analysis rejected it. Isn’t that what a journal like Analysis which is extremely selective should do on the basis of a negative report (even if you think that report was uncharitable?)

Philodemus
Philodemus
Reply to  Eh?
4 years ago

Yo. I was just cracking wise, but if you want some brief replies to your comments….

I suspect it’s perfectly normal, yes.

There was only one referee for my submission, who agreed with my main point, but still rejected my paper, partly because it was merely a reply piece (though I thought that was kinda the point of Analysis?). Also I didn’t engage enough with a particular philosopher who’s written a lot on the subject.

Seems like an R&R to me, but whatever. I don’t blame the editors for rejecting me. They have to make lots of cuts due to volume of submissions and also to maintain their reputation as being highly selective.

Eh?
Eh?
Reply to  Philodemus
4 years ago

Philodemus, fair play! Agreed on all counts

some person or other
some person or other
4 years ago

A minor point, but the use of ‘scientists’ to supposedly inclusively describe people in the humanities makes my blood boil. Philosophers are not scientists! Even if you disagree with this, I think you can at least agree that literary criticism is not a science! Why are we talking about science as though it is all that there is? I’m worried that this kind of thing contributes to the erasure of the importance of the humanities and embraces the “humanities must turn into science or die” model of thinking.

Kenny Easwaran
Reply to  some person or other
4 years ago

It’s just a problem that English doesn’t have the word “Wissenschaft”. We should all agree that all of us are engaged in collective intellectual pursuits. The question is what word there is generally for such a thing.

some person or other
some person or other
Reply to  Kenny Easwaran
4 years ago

Well, maybe (I mean, I agree that we need a word!), but I do think that this kind of word choice by individuals can contribute (even if in small ways) to a broader social shift/way of thinking about academic and intellectual pursuits that I think is deeply problematic/pernicious, and so it would have been better if they introduced a technical term, used ‘academics’ (I realize this is maybe not inclusive in a different way), or maybe just appropriated ‘wissenschaft’ (though I think that would probably come across a bit pretentiously).

Julia
Julia
Reply to  some person or other
4 years ago

Why not use “scholar” or “researcher”?
I do wonder whether there are relevant differences here between papers that report the results of empirical studies and papers that don’t. Most philosophy papers (and papers in other humanities disciplines) fall into the latter category. And even if a discipline has shared standards about how to evaluate such papers, it might take much more time for readers to apply these standards, that is, to determine whether a given paper is good. I don’t know enough about empirical research to be sure about this. But if this is true, it might mean that a system that works well for the sciences does not work nearly so well for philosophy and other humanities: it would take much longer for philosophers to read “unvetted” papers and determine if they are worth taking seriously.

Nicole
Nicole
Reply to  some person or other
4 years ago

Well, the turn away from considering all the disciplines a form of science and simultaneously a form of philosophy is historically new.

Aeon J. Skoble
Aeon J. Skoble
Reply to  some person or other
4 years ago

This jumped out at me as well. It’s at least possible that what might or might not work for lab researchers isn’t going to be the same for historical or philosophical research. If you get rid of peer review in the humanities, you can expect a lot more Naomi Wolf and Nancy McLean.

Trying to Earn Tenure
Trying to Earn Tenure
4 years ago

This would exacerbate an already extreme bias toward prestige and popularity.

The consequence of firing the gatekeepers is that we have a flood of new papers — a lot more noise in the system. Our attention is already taxed by the current rate of publication. It is incredibly difficult to keep up with new papers and books published in one’s AoS, let alone AoCs. So, how do you sift through the noise? You *could* randomly pick out new papers in your area, sure, but since we know there’s a huge range of quality in papers, this seems likely to be a waste of time. Instead, I imagine most of us would turn to reputation and pedigree.

If we had some reason to believe that the amount of time freed up by not having to engage in peer review would equal or exceed the amount of time required to read the extra papers, then this wouldn’t be an issue. But… that seems unlikely.

Kenny Easwaran
4 years ago

I’m unconvinced about the claims on gender dynamics. They say, “Insofar as there is gender bias—in the sense of women’s work being judged more negatively by peer reviewers—abolishing peer review will remove this and help level the playing field for men and women.” But it seems to me that this depends very much on the alternative. While it may be clear that there is some bias in the current system of gatekeeping, it’s not at all clear that the alternative model is less biased, rather than more biased. (Here it would be helpful to have empirical studies of mathematics publication.)

William Bell
William Bell
Reply to  Kenny Easwaran
4 years ago

Especially likely to be harmful to women if the prediction from the post above yours turns out to be true that:

> You *could* randomly pick out new papers in your area, sure, but since we know there’s a huge range of quality in papers, this seems likely to be a waste of time. Instead, I imagine most of us would turn to reputation and pedigree.

Douglas W. Portmore
4 years ago

What does it mean to get rid of (or abolish) peer review? I assume that there is no practical or legal way to prevent people from starting up or continuing to run journals that use peer review to help decide what to publish. And I assume that there is no practical or legal way to prevent people from submitting their work to journals that use peer review. So when people call for abolishing peer review, what exactly are they calling for? Are they just calling for everyone to voluntarily cease having anything to do with peer-review? I wouldn’t consider that getting rid of (or abolishing) peer review. But what else then?

Daniel Greco
Daniel Greco
Reply to  Douglas W. Portmore
4 years ago

Heesen and Bright can speak for themselves, but how would you feel about the following interpretation? A set of institutions–e.g., for distributing grant money, for hiring/promotion, etc.–where peer review doesn’t play a big role, and sites like arXiv do, is better than the set we’ve currently got.

You can then leave it as an exercise to the reader (or, the research community) how to get from here to there.

Douglas W. Portmore
Reply to  Daniel Greco
4 years ago

I’m not sure that we should spend much of our time or energy trying to figure out whether some other system is better than our current one unless we can see some practical way of getting to that possibly better system from our current one. And I don’t see what practical way there is to get from our current system to a system where there is either no peer-review or no decisions (about grants, hiring, promotion, etc.) based on peer-review from our current system.

Daniel Greco
Daniel Greco
4 years ago

I’m sympathetic to the worry expressed by “Trying to earn tenure”, though I think it applies more to philosophy (and probably other humanities), than to some of the sciences.

I think things like arXiv work best in fields where there’s a lot of consensus about both (a) which problems are important, and (b) what would count as satisfactory solutions to those problems. Under those conditions, I think it’s more likely that good work, once posted on a site like arXiv, will eventually get noticed and promoted, and moreover that the primary driver of whether research gets noticed and promoted is whether it is good by the standards of the field. That’s because people can read abstracts and get a sense of whether the paper addresses an important problem, and then if it does, they can read the paper to see whether the solution is satisfactory. If it is, you’ll recommend the paper to your friends, it will get a lot of downloads, and the primary purpose of publication–making knowledge public–will have been achieved. If everybody’s applying the same standards, then they’ll be disposed to read and recommend the same papers, and you should expect those papers to end up being widely downloaded/discussed.

The less agreement there is on (a) and (b), the less well I’d expect this sort of dynamic to work. Even if there’s agreement on (a), if there’s a lot of divergence on (b), then I might not bother reading a paper that addresses an issue I think is important, if my prior that it offers a satisfactory solution is low (whereas, e.g., in math, you’re a lot less likely to say you’ve proved P unless you’ve provided an argument that people will agree amounts to a proof). And if there’s divergence on (a), things are even harder.

It’s under those conditions–where you can’t expect “good” work to get noticed and promoted just because it’s “good”–that you’d expect the primary drivers of what does get noticed to be prestige and/or prior familiarity, in the way “trying to get tenure” describes.

I regard the above as a handwavy hypothesis, and I think it’s the sort of question where model-building social epistemologists (like Heesen and Bright!) could be able to offer much sharper insight; I’d like to see somebody try to build a model that captures the above, and maybe I’ll try one day if I learn to code. But my hypothesis is that in fields like philosophy, the signal of “philosophical value” people get from reading abstracts and papers is weaker than in math and physics, and that in such cases, you should expect the papers being read/discussed most to be more influenced by prior expectations about “philosophical value” than in fields where reading the paper provides more signal.

Nicole
Nicole
4 years ago

I thought this already existed? I believe they are called ‘blogs.’

Daniel Greco
Daniel Greco
4 years ago

In case the connection to peer review wasn’t obvious, the thought was that peer review improves the situation somewhat, by providing an extra signal of value. When standards are largely shared, that’s less important since popularity on a site like arXiv will be an excellent signal of value by shared standards. But where standards are less shared, popularity will be a weaker signal of value, and there’s more of a role for peer review.

Dale E Miller
4 years ago

Can we just get rid of reviewer 2?

More seriously, I think that one consequence of abolishing peer review might be to further entrench a “star system” in which the work of many scholars attracts even less notice than it does now. Peer review may privilege some groups relative to others, but scholars who are not privileged—those not at R1s, women, faculty of color, independent scholars—all still have real opportunities to be published in good journals. And when this happens, the work will get at least some attention. If we abolish peer review, then the playing field may be completely level in terms of making work available; we can all post to Philpapers. However, the work of anyone who has not already made a name for herself will be much less likely to be noticed.

Paul
Paul
Reply to  Dale E Miller
4 years ago

Yes. Relatedly, I can’t see how “the effects of actual and expected gender bias in peer review would be reduced”. A typical feature of the peer review system is its being ‘blind’, such that the author’s gender etc. is unknown to the reviewers. Presumably this would be lost under the alternative proposed here.

Friedrich Wilhelm Grafe
Reply to  Dale E Miller
2 years ago

first thoughts …

Peer review may continue to serve as an admission ticket to journals, but its perhaps the time to dismiss both: anonymity of preprint authors and anonymity of reviewers

Putting preprints on some open access server for review, scientific public may have multiple evaluation options:

  1. non anonymous review statements by peers
  2. joining a discussion of the paper with the author
  3. factual voting by (dis-)likes and/or downloads
  4. … other … ?
Arthur Greeves
Arthur Greeves
4 years ago

Yep, and we should get rid of that absurd insistence that the FDA test drugs before use, too. The people harmed will be so happy that the drugs got to them faster.

Maja Sidzinska
Maja Sidzinska
4 years ago

Or we could have referee work count as a formal/quantified/paid part of scholarly work, and have it count for purposes of promotion/tenure. That would get rid of the opportunity costs.

I am very doubtful about the other benefits, for the following reasons.

• Scientists would be able to share their discoveries more quickly.
**part of what peer review does is ensure that what is claimed as a discovery actually is one; there is a bias against negative results and peer review limits this bias by forcing or at least encouraging scholars to contextualize their work; false positives are limited this way**
• More than just the privileged few who are enmeshed in particular research networks would have early and free access to scientific findings.
**it is not clear how self publication would democratize access to ‘scientific findings’ in a meaningful way; the most serious barrier is not lack of membership in a research network, it is lack of access to libraries and/or technology/computers; in other words the more important difference is between those who don’t have access to education versus those who do, rather than the difference between those who don’t have access to research networks and those who do**
• Scientists would be freer in their choices about how to spend their time, rather than being “conscripted” into reviewing manuscripts.
**Conscripted? I find it odd that this is seen as conscription; I thought engaging in scholarship w/ others was the point? Peer review is the moment of deepest engagement. And if it was counted and remunerated as a regular part of work, then it would not present an opportunity cost. On a more psychological level, I’m not sure that the motivation to ‘be heard’ (in the form of putting out research) would lead to better outcomes than the motivation to ‘be interrogated’.
• The effects of actual and expected gender bias in peer review would be reduced.
Er, if pre-publication peer review were eliminated then yes, gender (and other) bias in pre-publication peer review would be eliminated. How this would improve the overall problem of various biases infiltrating the publication/scholarship process is not at all clear.
• There would be significant savings in library budgets.
We should support robust library budgets precisely because this would democratize access to research in a way that takes into account the most serious barriers to access. Rather than asking how libraries can save money, shouldn’t we be asking how to better fund libraries?
• When it comes to assessing researchers, the role of the “short-run credit” of publication in a high-impact journal may be reduced, while that of the “long-run credit” of citation metrics may be increased.
Long-term citations fundamentally mark popularity not quality; there may be a relation between popularity and quality, but then again there may not. Same for publication in a high-impact journal. That so-called ‘long-run credit of citation metrics’ is better cannot be assumed however.
• The power of gatekeepers would be reduced, making the evaluation of research a “more in line with general communal norms accepted within science.”
Pick your gatekeepers. Who is gatekeeping what is political. If you get rid of peer review, you’ll simply get a new set of gatekeepers.

I am very skeptical of this proposal.

Ned Hall
Ned Hall
Reply to  Maja Sidzinska
4 years ago

Maja: I think it would be *brilliant* to find a way to count referee works for purposes of promotion. Doing so would give junior faculty a reason not to avoid refereeing like the plague. And, depending on the details, it might give all referees a strong incentive to be both timely and constructive. There is at least one site (publons.com) that tracks refereeing work, though I have no idea whether it’s had much of an impact in philosophy. Anyway, really good suggestion; might be helpful for all of us to think through how best to implement it.

Maja Sidzinska
Reply to  Ned Hall
4 years ago

I am a afraid that academy-wide culture/labor/structure change would be required to fully implement my suggestion. Perhaps incremental change would be doable however, maybe in the form of firstly counting refereeing as promotion-relevant. Would one do this by lobbying various higher-ed governing bodies (country-specific)? Or by urging discipline-specific professional associations (e.g. the APA in the US) to use their influence? (Not sure these orgs have the kind of influence required.) Or institution by institution? Or is this a lost cause since the humanities are firstly trying to simply survive, and since a capitalistic ethos now governs them (and higher ed institutions in general)?

I am just a grad student (have MA, starting a PhD program in fall) but I have some experience in publishing. It has struck me as very unfortunate that what I see as perhaps the most important philosophical work (deep critical engagement as occurs in the refereeing and other feedback-giving process(es)) as being precisely the work that educational institutions don’t count. But they SHOULD. Because I think (naively, to be sure) that that is what educational institutions are FOR. Yes, I realize this romanticizes the academy.

If actual professional philosophers begin any projects for the purpose of exploration/ implementation, I would love to be involved and support such projects.

Matt Weiner
Matt Weiner
Reply to  Maja Sidzinska
4 years ago

“part of what peer review does is ensure that what is claimed as a discovery actually is one”

This seems to me like less of a concern in philosophy than in physics and other sciences (or empirically based humanities fields such as history). In other fields, I take it that when something is published it’s sort of “entered into the record” and other scholars are at least to some extent entitled to rely on the truth of the factual conclusions established therein, if not on the interpretations. (I may be wrong about this and am almost certainly oversimplifying.)

But in philosophy that doesn’t happen–just because a result is published doesn’t mean people are going to rely on its truth. To put it mildly.

One thing I think this supports is that we should try to lower journals’ rejection rates (which means finding a way to publish more papers in total). If only 5-10% of submissions are accepted, there’s not much harm done in accepting the next 5-10%. And

“Er, if pre-publication peer review were eliminated then yes, gender (and other) bias in pre-publication peer review would be eliminated. How this would improve the overall problem of various biases infiltrating the publication/scholarship process is not at all clear.”

This seems very true. If we want to replicate the benefits (if any) of blind reviewing here, perhaps one proposal would be to have submissions to the philosophy archive remain anonymous for a period of time?

“We should support robust library budgets precisely because this would democratize access to research in a way that takes into account the most serious barriers to access. Rather than asking how libraries can save money, shouldn’t we be asking how to better fund libraries?”

Perhaps, but we can do more about changing the publication model of our discipline than we can about making people fund libraries more. And anything that decreases the amount of money that publishers drain from the system is good. Especially for-profit publishers.

Maja Sidzinska
Reply to  Matt Weiner
4 years ago

“But in philosophy that doesn’t happen–just because a result is published doesn’t mean people are going to rely on its truth. To put it mildly.”

–it’s true that this is less of a concern in non-empirically-heavy disciplines, but even in philosophy I see a lot of positive claims being put forward that don’t answer to either existing or foreseeable concerns. Professional philosophy also has an appetite for positive arguments, and I’m not sure it should (although perhaps it should, I’m not sure).

“This seems very true. If we want to replicate the benefits (if any) of blind reviewing here, perhaps one proposal would be to have submissions to the philosophy archive remain anonymous for a period of time?”

–Again I’m not sure about the answer. I just wanted to point out that one process will replace another, so getting rid of bias in one process does nothing to get rid of bias in any process that replaces it (and I am sure that processes, left to their own devices, are biased since bias is just ‘in the water’). Some others have pointed out the same thing in the comments. Contra some people, I don’t think that eliminating (blind) peer review would ultimately help address representation problems.

“Perhaps, but we can do more about changing the publication model of our discipline than we can about making people fund libraries more. And anything that decreases the amount of money that publishers drain from the system is good. Especially for-profit publishers.”

–Of course library funding is only tangentially related to the questions/issues being addressed here. But I think I see the whole academic/higher ed/publishing *system* differently. For me, the ideal is: a generously and fully publicly funded academic system, where the work of publishing is seen as work and is remunerated accordingly. (Although I worked at an enormous corporate publisher and I think such publishers in general should die; there is publishing and there is publishing; it’s not all the same; corporate interests/models should not be guiding academic publishing.) For instance, I think all college and research should be publicly funded, all academic publishing and all libraries should be publicly funded. I think many initiatives that are designed to solve problems that result from the above not being publicly funded actually make the problems worse in the long run, e.g. open-access mania; free-course materials mania; these ‘solutions’ don’t address the underlying problems–that editing work is being done for free (else not done); that all student materials aren’t publicly funded, etc.; these just make academics (over)work for free. Which is the problem that motivated the suggestion of getting rid of peer review in the first place. I see the suggestion to get rid of peer-review as a ‘band-aid’ fix for huge structural academic problems relating to labor and education. And I just don’t think this will deal with the root issue, which is what needs to be dealt with, otherwise we will simply see new symptoms of the same old disease. E.g., we get rid of peer review and then face increased pressures/expectations to ‘produce’ research… I think we know where this would lead… a race to the bottom.

Matt Weiner
Matt Weiner
Reply to  Maja Sidzinska
4 years ago

Thanks for the reply!

“even in philosophy I see a lot of positive claims being put forward that don’t answer to either existing or foreseeable concerns.”

I’m not sure if we’re talking past each other a little here–I was not very explicit in my original comment and I’m not sure I’m talking about the exact same thing that you are here. (It’s very muggy here and I may not be thinking too clearly.)

What I meant, to spell it out, wasn’t that philosophers don’t put forth positive arguments for P. is just because some philosopher publishes a bad paper arguing for P, that doesn’t mean that other philosophers are going to take P for granted in their papers. And if someone else does put forth an argument that depends on P, they’re more likely to be downgraded if the original argument for P sounds bad or implausible. (The “This paper seems to depend on a commitment to view X but who cares about view X?” kind of objection.) So I don’t think a bad conclusion that P would propagate as much in philosophy as other disciplines.—There is, though, the risk that the bad paper for P would generate unnecessary papers about “Why Y’s awful argument for P is awful.”

“I think such publishers in general should die; there is publishing and there is publishing; it’s not all the same; corporate interests/models should not be guiding academic publishing.”

*enthusiastic applause*

“I think many initiatives that are designed to solve problems that result from the above not being publicly funded actually make the problems worse in the long run, e.g. open-access mania; free-course materials mania; these ‘solutions’ don’t address the underlying problems–that editing work is being done for free (else not done); that all student materials aren’t publicly funded, etc.; these just make academics (over)work for free.”

Great points. I think it’s a vexing question as to how much we should push for broader systemic reform and how much we should just try to work within what we have. Sometimes I think it’s a question of pushing around the work–someone’s got to read and think about the papers sometime–and one thing about the current system is it seems like there’s a lot of duplication of work. If a paper gets sent to five journals before publication, which seems like not an unreasonable number, that’s something like ten people who review it, and a lot of that work may wind up getting duplicated.

Thanks again for the response.

David Wallace
David Wallace
4 years ago

As the authors note, this is basically the model that has been used in (large parts of) physics for twenty years (and to some degree is also the model that philosophy of physics uses). It’s a good model: I’ve advocated it many times on this and other blogs. But precisely because it’s been around for twenty years in a major scientific field, it seems to offer a test bed to see whether the authors’ predictions are correct – and for physics at least, many of them don’t seem to be correct.

Going through the authors’ list of advantages:
– Sharing scientific results: yes, the physics model is great for this, both because results are available much more quickly and because arxiv is open access. This is the main reason I like the physics model (and the main reason I’m skeptical about over-prioritizing double-anonymous review when it clashes with arxiv-style dissemination).
– Freeing up scientists’ time because they don’t have to do peer review: has not happened at all so far as I can tell. Peer review is still hugely important in physics and people still spend lots of their time doing it.
– Reducing gender bias: I don’t know about this, it’s a difficult data problem. I’d be surprised if it’s made a difference, partly because I don’t find the evidence that peer review has a gender skew to be very persuasive, but that’s just my own judgement.
– Saving library resources, because all journals will be forced to be open access: not true. Decades after arxiv, the main physics journals remain subscription-based and closed-access. Maybe that’s starting to change as a consequence of the broader open-access movement, but arxiv itself hasn’t forced it. (If anything, it may have reduced the pressure, since you can *normally* get a recent article off arxiv in any case.)
I have to say that I’m surprised that this is the case, especially given that the typesetting of an arxiv article is usually almost identical to the final publication – but empirically it is the case.
– reducing the ‘publish or perish’ culture… well, put it this way: if you have friends who are post-docs or late-stage PhD students in physics, and you want to keep them as friends, I don’t recommend congratulating them on getting into a field that doesn’t have a publish-or-perish culture.
– disappearance of journal rankings and impact factors as a factor in hiring and promotion: I am pretty sure these have not disappeared, but I haven’t done a lot of hiring in physics so I could be wrong.
– reducing the power of gatekeepers: it’s probably true that journal editors have less impact on the shape of physics than they used to, and that’s probably a good thing.
– runaway Matthew effect: well, this definitely happens in physics (for better or worse) but I don’t know how much that’s to do with the arxiv model. (And it’s worth realizing that the model, at least in high-energy physics, actually precedes the internet, so you’d need to go back a fair way to assess it.)

It’s difficult to know how much this would generalize to other subjects. But it’s interesting that at the dawn of the arxiv era (basically when I was a physics PhD student) everyone was talking about how it would mean the end of the traditional journal and would bring about a revolution in peer review. That this hasn’t happened, two decades later, makes me think that the traditional model has a lot of resilience, even in the face of open access archives.

Sideshow Bob
Sideshow Bob
4 years ago

“By the way, I’m aware of the irony of appearing on TV in order to decry it, so don’t bother pointing that out.”

Outlander
Outlander
4 years ago

My general sense in my discipline is that the peer review process itself tends on average to only modestly improve papers, and it occasionally makes them worse and occasionally makes them much better. Measured against the time and energy costs, that might not seem worth it. But the existence of the peer review system itself likely motivates scholars to be more careful, anticipate criticism, check robustness of conclusions, etc – it certainly motivates me, and that seems like a tangible benefit (as much as peer review/reviewer 2 drives me nuts sometimes)

Yulia
Yulia
4 years ago

I’m surprised at the number of negative comments. It sounds very conservative, as if the existing system is the only option that should be preserved forever. Meanwhile, the biggest problem with peer review is that in our age it is not “peer” anymore. In many cases reviewers are not who they are supposed to be, such as graduate students (“your work has been reviewed by experts in the field”, yeah), or at best they are two-threr scientists who may not even be specialists in the area in question! In no way this is “peer” review anymore. Crowd-reviewing, open to everyone, would make much more sense. Yes, it’s a problem that we don’t have a system like that but that means we should try to think of a system, not blindly to preserve status quo.

Dale E Miller
Reply to  Yulia
4 years ago

How would crowd-reviewing work if crowds never read a paper in the first place? We know that most journal articles are read by very few people now. In a world without peer review, my fear is that most articles that weren’t written by big names would never be read at all. So even if a paper makes a very important contribution, if it’s not from a star it will go unnoticed. With peer review, if a paper is published then potential readers at least know that some people have already looked at it and considered the quality high.

David Velleman
Reply to  Dale E Miller
4 years ago

There are ways of solving this problem — e.g., an reviewing economy in which the right to post an additional paper in the archive must be earned by rating and reviewing several previously un-reviewed papers that are already in the archive. There may also be ways of crowd-sourcing expertise, by allowing reader ratings not only of papers but of reviews, as on some commercial websites that ask “Was this review useful?”. A participant’s ratings as both author and reviewer could be used to weight that participant’s ratings and reviews in the composite score of a paper. There might also have to be a way of tagging both papers and participants by their field. (This is a *very* rough description of a system that would take a lot of experimentation to refine.)
The main obstacle to crowd-sourcing peer review, as I see it, is the difficulty of preventing friends from rating and reviewing their friends’ papers and reviews. The submitters of ratings and reviews would *not* be anonymous, but the authors of papers would have to be anonymous, at least initially, in order to prevent gender, race, age, and reputational bias. Maybe the author of a paper would remain anonymous only until M reviews of that paper had been entered — whereupon, the author’s names and affiliations would become public, allowing conflicts of interest to be detected.

Dale E Miller
Reply to  David Velleman
4 years ago

It’s an interesting model, although if the reviews are going to be public then wouldn’t attaching the names of the reviewers call the impartiality of the reviews into question? Reviewers might not be afraid of writing negative reviews, since presumably they could simply choose not to review papers that they thought were weak. But they might have some perverse incentives to write positive reviews. If the reviews would themselves be rated, then perhaps it would be sufficient to note that, e.g., “This report comes from a five-star reviewer.”

David
David
Reply to  Dale E Miller
4 years ago

I am preparing a detailed proposal to submit to the APA blog, and (eventually) to the APA itself for funding.

David Wallace
David Wallace
Reply to  David Velleman
4 years ago

Anonymity of papers seems to get in the way of speedy dissemination of results. (Certainly, given the choice between submitting to an anonymous archive and one where I can include my name, I’m going to do the latter.)

Paul
Paul
4 years ago

“The power of gatekeepers would be reduced, making the evaluation of research a “more in line with general communal norms accepted within science.””

I’m not convinced of this either. I can imagine a system of postpublication peer review giving political operatives a bigger opportunity to mobilise against heresy, identifying works (or authors) they don’t like and exerting influence over journals to ensure those works (or authors) get picked up.

Paul
Paul
Reply to  Paul
4 years ago

*to ensure those works (or authors) DON’T get pick up.

Sign 'em!
Sign 'em!
4 years ago

This has probably already been floated, but worth emphasizing: Won’t a lot of the BS be fixed by making one simple tweak to the process: require that referees SIGN their referee reports? Adding that extra feature of accountability would (based on common sense psychology–of course, would need studies to confirm, etc.) very likely cause referees to give verdicts they’re willing to stand by in the light of day with others looking. This will plausibly have two helpful upshots: fairer verdicts (i.e., fewer handwavey rejections, and handwavey acceptances), but also more thorough reports. This would slow things down a bit, perhaps, as reports would take a bit more time. But, the process itself would be much fairer. Thoughts?

Matt
Reply to  Sign 'em!
4 years ago

There is a bit of irony in a blog comment – something of little importance that no one is likely to take offense to or bother the author about – calling for referee reports to be signed, but then being posted anonymously. But on the substantive point, I do a fair amount of refereeing – at least 10 papers a year, sometimes more. I’d certainly do significantly less if I had to sign them. I am not an overly harsh referee, I think, but if I had to worry about people contacting me and lobbying me about my comments, or retaliating against me in different ways, or so on, it would make the refereeing process both more stressful and also much more time consuming. Rather than providing comments and opinion, I’d feel obliged to provide something much more like a reply piece. That might be good for authors, but it would take so much more time that I doubt I’d do more than 2 or 3 reviews of papers a year, even leaving aside the worries about personal attacks. Given this, it seems like a pretty bad idea to me.

Marc Pinsonneault
Marc Pinsonneault
4 years ago

I work in a different field (astronomy and physics), so I don’t usually comment. But the astronomy example is quite useful, as David noted above. It’s also quite different than replacing peer review.

First of all, there are still gate-keepers. You have to be “in the system” to be able to post papers to the public online archives. Crank submissions are suppressed via formal and informal mechanisms. You can post articles not submitted to journals, but not many people bother to read them (and many are filtered out as irrelevant). Like online blogs, publication archives have to be managed and decisions have to be made.

There are two philosophies: you can post on submission or post on acceptance, and which you do varies by discipline and sub-field. A small minority of journals, like Nature, have pre-publication embargoes. The journals survive on a combination of author page charges and support from scientific societies, in addition to subscriptions (which are pretty obviously devalued in this approach.)

This model removes many of the barriers involved – access to articles without subscriptions, allowing ideas to disseminate without being suppressed by referees, and so on. However, journal acceptance is still highly valued, and the refereeing process is still essential. Note that some journals in my field do permit non-anonymous refereeing, but it has real headaches associated with it: some authors are vindictive and resent criticism, just as some referees do. The editor can control the latter (by ignoring bad reports or soliciting different ones), but the former can lead to professional and personal retaliation that can be very difficult to manage. Anonymous refereeing is not a thing to give up lightly for that reason.

Basically, if you just want to publish things on the internet, you need some system to filter out junk, and some way to endorse papers and indicate that they’re worth reading. If there is a working better alternative it hasn’t manifested itself yet. I doubt that the self-selected set of internet commenters on papers, for example, will give better advice that referees solicited by competent editors on average. But I could be wrong!

Alexander Bird
Alexander Bird
4 years ago

Many of the problems of peer-review in science — and other problems such a publication bias — arise because the reviews are carried out after the research has been done, rather than before. Increasingly, however, journals are publishing Registered Reports. Here the researchers submit a protocol with detailed descriptions of the research to be carried out and of the methods to be used in analysing the results. Peer-reviewers comment on the protocol, and if the reviews are positive, the journal will agree to publish the paper (report) that results once the research has been carried out (subject to everything been done as the accepted protocal claimed it would be). A major advantage of this approach is that it allows the reviewers to help the researchers improve their methods before carrying out the experiments (or other investigation) (this adds to the motivation of the reviewers to do a good job — it will make a difference to the science that is carried out). Another is that it reduces the temptation for scientists to engage in p-hacking, HARKing and other questionable research practices. And finally it focusses attention on the quality of the research ideas and methods, not on whether the outcome is dramatic. Null results will be published, avoiding publication bias.
In my view, a disadvantage of the proposal of Heesen and Bright is that it underestimates the difficulty and effort required in assessing the quality of research, especially technical aspects such as the statistical analysis. If there isn’t gatekeeping on this, then anyone who reads any scientific paper posted on a server must make these judgments themselves. That would involve a lot of duplication of effort. And furthermore, given division of labour and specialisation of expertise, many scientists will not be able to do this for themselves. While Heesen and Bright present a lot of evidence that prepublication (but post-research) review does not help improve quality as much as one would want, the reasons for that fall away for registered reports.