The Case for a Peer Review Market (guest post)
“The academic peer review system as it currently stands is frustrating and dysfunctional for many of those who participate in it.”
So writes David Thunder, Research Fellow in Political Philosophy at the University of Navarra.
In the following guest post, he briefly identifies some of the problems with the current system of peer review, and suggests they can be remedied with a kind of referee marketplace in which editors can shop for—and purchase—referee services.
[detail of an artwork by Yau Hoong Tang]
The Case for a Peer Review Market
by David Thunder
The academic peer review system as it currently stands is frustrating and dysfunctional for many of those who participate in it. Below, I detail some of its most salient limitations, and afterwards propose an innovation that could mitigate these issues.
Limitations of the Current Peer Review System
1. Because peer review services are often pro bono or done for a nominal fee, they rely on the goodwill and sense of personal responsibility of each reviewer, not on an enforceable contractual obligation. Because of the pro bono nature of many reviews, the motivation for doing them is something like “duty to the profession,” unless a reviewer has a strong personal interest in a specifical manuscript. There is no enforceable contractual obligation to speak of. This has two negative consequences:
- First, reviews may be either half-hearted or submitted very late, which has negative knock-on effects for publishers and authors alike.
- Second, due to the limits of moralistic motives that are not remunerated, editors may have to spend months trying to secure a scholar willing to conduct a time-consuming review.
2. Because there is almost no form of public accountability for reviewers, and they know this, the quality of reviews is mixed. Some are excellent, others acceptable, and others based on personal prejudices or superficial readings of a text. In any case, the effect of a pro bono system, combined with the fact that reviewers’ work is not publicly evaluated or held accountable, is to create a class of gatekeepers who only answer privately to editors for the quality and timeliness of their work, and cannot realistically be pressured too much given that their work is pro bono.
Effects of the Current System for Authors and Publishers
1. Publishers are put in a difficult situation in which they subject authors to lengthy, career-damaging delays over which they have little control.
2. Publishers may find that their ability to bring work to the market efficiently is undermined by needlessly drawn out review times.
3. Authors may find articles and book manuscripts sitting under review far beyond the estimated review times. In the case of book manuscripts, these delays may be especially long (anywhere from 6 months up to 18 months), and have negative repercussions for a scholar’s career.
A Proposal for a Peer Review Market
I propose to create a virtual peer review market, through which both authors and publishers/editors can search for available peer reviewers and solicit their services on a competitive and professional basis.
The basic idea is that each peer reviewer compiles an online profile which may be independently vetted before it goes live. The profile would include name, educational background, publications, number of peer reviews conducted, average rating for their reviews, and a negotiable offer price and guaranteed turnaround time for reviews.
Editors and authors would also have an online profile in the system, and they could solicit peer reviewers’ services and indicate their required turnaround time. They could also use the system to rate the work done under different variables (e.g. quality, comprehensiveness, punctuality), and make payment for services rendered.
Benefits of a Peer Review Market
People who are paid for the work they do tend to give it a higher priority and feel a strong sense of duty to do it properly and on time. In addition, if reviewers know they will be rated publicly for their work, they are more likely to take is seriously. The end result of a marketized system is more peer reviews of a higher quality, and more peer reviews turned in on time.
A second advantage of a public peer review market is that editors and authors may search for eligible reviewers from a much larger database that includes important data about past experience and reliability as well as areas of specialization. When people know they can make money from reviewing manuscripts, they will have a greater incentive to up their game and develop an attractive portfolio of review experiences. This is a win-win for authors, editors/publishers and reviewers alike.
Challenges and Questions
A professional peer review market may pose certain challenges, especially at the start:
- Universities, journals and book editors would have to expand their budget for academic reviews if pro bono reviews are phased out.
- Some unscrupulous reviewers may try to “game” the system by getting paid for low quality reviews. But this should be discovered fairly quickly when their work is rated by editors and/or authors.
What do you think are the strengths and weakness of this proposal? Do you have any practical suggestions for the implementation of this or other reforms of our peer review system?
“if reviewers know they will be rated publicly for their work, they are more likely to take is (sick) seriously.”
Or they will not review at all.
I think this would just increase the work load for everyone. As a reviewer, I would have to create yet another online profile (in addition to my profiles at Researchgate, Academia.edu, Philpapers etc.). As an editor, I would have to “review” the review as well and provide some sort of rating to the reviewer.
There is a simple solution to the “problem”. Just accept the referee invitations and write the review when you are asked to do so. It is often enough if the reviewer says whether the paper is any good or not. No need to write as long reviewer reports as the paper itself (this happens surprisingly often).
Cool idea, but some initial concerns:
(a) that the actual “market rate” would be extremely low. If people are doing X for free it might be because nobody really wants to pay for X-ing, e.g., nobody is going to pay me to raise my own kids. So, it’d end up with some random internet entrepreneurs in an underdeveloped country doing the reviews.
(b) maybe not everyone feels like this, but I find reviewing an essay to be something quite “intimate”. It’s not the kind of thing where you want everything you say to be in a spotlight and recorded for posterity. So, I think that reviews would end up becoming more “stiff” and formulaic, less spontaneous and freewheeling.
One possible downside is that extrinsic motivations can crowd out intrinsic ones, so that some people who would do a task for free won’t do it if they’ll get paid. See https://en.wikipedia.org/wiki/Motivation_crowding_theory for a quick summary of the idea. Whether and how this would apply to paper reviewing is an open question.
I wonder if this sets up a perverse incentive for reviewers to go easy on papers to get higher-quality ratings. Something similar is happening in teaching with student evaluations. Professors are incentivized to go easy on their students because they know this generally correlates with more positive evaluations.
The institutional check on this might be the editor independently vetting the quality of review reports, but I don’t think they can be expected to do this well. After all, papers often involve niche issues that only a select group of reviewers knows well. This is surely at least one of the reasons editors rely on other peer-reviewers to verify the quality of the papers.
That incentive does not strike me as obviously perverse. It might even be a good thing to incentivize.
That incentive strikes me as obviously perverse. It might even be a good thing to disincentivize.
I’d assumed that the judging of review quality, timelines, etc., would all be done by editors, in this system. Reviews are for editors, not for authors, and it’s editors who have to decide which reviewers to use, so it’s the judgment of their fellow editors that they’d care about.
I don’t think editors have any reason to reward unduly positive reviews.
This seems like a good idea if we want the system to continue in the same direction: allow market forces to have even more influence on the discipline. Alternatively, one could ask: why are there so many new submissions (from MA and PhD students, and from junior faculty), and why are there not enough reviewers? Is publish-or-perish a good idea? What does this principle (PoP) do to those who are subject to it? What does it do to the quality of papers? Adjustments to market forces (this is what the proposal is) make the machinery run more smoothly, but they don’t address the underlying problem.
I want to gently challenge the presupposition of the proposal here that the system is broke. Given the way things currently are (e.g., very very hard to get in the top 5s, which are the gold standard) the cream does seem to rise to the top.
Perhaps. But of those five journals, two are run by the same editor and for the last decade have stopped accepting submissions for close to half of the year, and the other three are documented to have undergone a shift from a pluralist-inflected pragmatism to a narrow analytic focus from the middle of the last century. So even if the cream is rising to the top, it looks like we’re skimming off only part of it.
Sorry, that should be pragmatist-inflected pluralism. Joel Katzav documents this for the Journal of Philosophy in “Analytic Philosophy, 1925-1969: Emergence, Management and Nature”, while he and Krist Vaesen make the same case for Mind in their “Pluralism and Peer Review in Philosophy”, and for The Philosophical Review in their “On the Emergence of American Analytic Philosophy”. Here they are at p.774 of the last essay:
I disagree that the cream is rising to the top; certainly, some of it is, but saying that it is hard to get published in these journals does not mean that everything that is good enough is.
I’m confused. What is being asserted here, that, by virtue of being very hard to get into, the “cream”, all the best publications in philosophy, appear in these five journals? Surely not.
Hi Margaret, the reasoning is more like this. People keep saying ‘the system is broken!’ I say: cool your jets – a (strong!) defeasible indicator of any peer review system’s ‘not’ being broken is that the cream is rising to the top in that system. Premise: it is rising to the top (in our current system) – one that in fact puts a halo over the top 5. So we have a defeasible indicator that this (top 5 halo) system is not broken. Preston above has challenged the cream premise. To the extent he’s right, my reasoning doesn’t go through. I’m just saying here what the reasoning is.
Good enough. I myself am sympathetic to the view expressed by Another Philosopher. Indeed, given how random the peer review system is, it seems to be inevitable that the “cream” will be widely scattered.
‘Cream rising to the top’ is the wrong metaphor. It’s more like ‘flavour of the month being promoted’.
In theory this seems like a great idea. Morally refereeing is valuable work and it should be compensated in some way. Practically, it seems like basic economics. Pay more for something (in this case pay at all) and supply goes up. But I don’t see how in practice this won’t either be a complete and utter disaster or, at best, do nothing to solve the problem. How much money would we need to pay referees to make much of a difference in their behavior? I guess some people who are on the line about taking a review might do it for say $25 but very few will. When you consider that a minimally decent review for most papers takes at least three hours, then that’s below minimum wage. And as Tim O’Keefe points out extrinsic rewards can crowd out intrinsic motivations so as weird as it sounds there are some people who will do a review if they think of it as a service to the profession or paying what they owe to the peer review system who won’t for $25 or even $50. So if the compensation isn’t high enough this might actually make things worse since it will reframe this in a way that makes requests easier to turn down for many of us. How much money would you need to take on three hours of work you otherwise wouldn’t have much motivation to do? I guess it depends but for me I definitely wouldn’t do it for less than $100 and I’d probably want more like $150. Well if we have two referees per paper that’s $300, where is that going to come from? I think some submission fees are fine, but if they’re $300 or more per paper that’s going to create massive problems. Adjuncts will simply be priced out of publishing.
Then there’s the perverse incentives this creates. Editors already put their fingers on the scales in all kinds of subtle and not so subtle ways. This will make that much much worse. It’s hard to stick to your principles if you’re just annoying an editor by not okaying a paper they want to publish. Now imagine how much harder that will be if by doing so you might cost yourself hundreds or thousands of dollars a year in income.
Ultimately I agree with Miroslav Imbrisevic and probably go further. The crisis in philosophy publishing is a symptom of much deeper forces and only addressing them will do anything to solve it.
My evil twin, not me, asked me to post this: “For what it’s worth, I think getting paid $50 a pop would be enough to significantly influence my behavior. Mostly my timeliness: I do a fair amount of reviewing, but reviewing deadlines are a relatively low priority since I’m not getting paid for or evaluated on it. But I vaguely enjoy reviewing and think it is important, so if I thought my tardiness was going to impinge on my ability to review in the future–and make $50 a pop doing so–I think I’d be significantly more timely with my reviews.” Again, that was my evil twin, don’t bother chastising me.
I agree it would influence my behavior though only a little. But keep in mind I’m someone who’s inclined to accept requests anyway (keep that in mind if you’re an editor reading this I suppose) as it seems you are too. I was pretty much Buridan’s ass with one request to review a book proposal and the relatively modest payment pushed me over. But if I wasn’t inclined to do it? Eh I doubt. And to do much to solve the problem we need to sway those people who aren’t inclined to review in the first place. I also think there are cheaper ways to insure promptness. The editor of the BJHP for instance just sent me a polite personal email like the day I was over their two month deadline. Knowing an actual person was checking in got my butt in gear. I mean honestly I think paying referees some modest amount is probably a good idea if for no other reason a recognition of its value (journals could and should suggest TT faculty forego theirs like the LARB does for contributors who are financially stable). But I’m incredibly dubious Thunder’s suggestion for large scale change will deliver what it promises.
Something that I have wondered about is whether we couldn’t just found a journal that accepts papers which the author can verify have had at least two split decisions. A single paper can receive two back to back split decisions, which means that two reviewers both thought it was good enough for publication, and not be published anywhere. This is totally the luck of the draw, as it was just as likely that the first or second journal could have asked those two reviewers and the paper be accepted unanimously. Papers like this will get sent out a third (or more) time and will require additional reviewers and thus clog up the reviewer system while already having proven themselves worthy. It doesn’t seem like there is a point ti this. Why not just have a journal that recognizes that these papers are worthy and are only not published out of luck, and then publish them?
Of course, you would need to verify with the editors that the reviews were genuine, but that seems doable.
I don’t really understand your workings here. We would want to take into account the number of rejections it had received as well, no?
Another thought is that peer review is unfixable. There are too many journals, too many submissions, and too few readers. The whole structure doesn’t even make sense, and basically only exists for people on the job market and those applying for tenure and promotion. (Right? Who else is seriously invested in it?)
So we don’t really need more reviewers, so much as fewer journals and fewer submissions. Or a different model altogether that somehow moves away from peer review altogether (or mostly/partially).
What would such a model look like? Maybe something more like a pre-print archive. Just upload stuff. Conference it about and see what people start to read or get excited about. It’s less of a “market-based approach” than an “ideas-based approach”. It barely requires reviewers (or publishers). How does anyone get hired or promoted without acceptance rates? Well, I guess the quality of their work would have to stand by itself (or be corroborated by the community).
I’m just skeptical that we’re going to make much progress on this very broken system, especially by tinkering about the edges of it.
Certainly some other fields have pre-print archives. But they also have journals. I’d like to know more about how these two ways of getting ideas out there are related, and what role they play in things like hiring and promotion. Can anyone here speak to that? E.g., if a mathematician archives a good proof of an important theorem, and it’s widely recognized that this is so, does it actually matter to anyone whether it’s published *in a journal*? If so, why? (Not a rhetorical question.)
I understand that journals, and concepts like “top five journal,” allow people to outsource their judgements of what philosophical work is good and what isn’t. But is that the sort of thing that one should outsource?
A related thought:
It was pointed out in this earlier DN post that books are generally reviewed post-publication. Could a post-hoc review work for journal papers? For instance, since most/all journals are also online, it shouldn’t be hard to append those reviews, responses, criticisms, etc. to the original paper as an audit trail of sorts.
(Though some book manuscripts do go out to external reviewers prior to publication, this seems to be the exception rather than the rule. And they also seem pro forma, unless the publisher is looking for an excuse to reject it. Certainly, a book manuscript that’s 10x the length of a journal paper isn’t getting a review that’s 10x as long, as far as I can tell.)
In general, it seems crazy (to some folks, anyway) to think that we should want to give 1-3 random reviewers the power to gatekeep and prevent an entire book or journal paper from being published. That’s not “peer review” or anything that speaks for entire disciplines, but it’s only a jury of a few randos with unknown motivations, idiosyncrasies, etc. They might not even be qualified, judging from some of the review requests I get. So, the risk of sampling error and bias in this small jury pool is huge.
The Nordic Wittgenstein Review has taken a step towards this model, whereby all papers accepted by the normal process also “undergo an additional open peer review” where “all registered users will be able to comment on and to discuss the accepted articles online for a specified period.”
I believe that during this period, the author has the opportunity to amend their paper in response to the comments.
Now, I don’t know what their experience has been and this model is not fully what you have in mind, Patrick, but to me this suggests the following system:
When a paper is submitted to a journal, the editor finds a reviewer for preliminary review which would basically be a sanity check. The paper then undergoes open review after which the editor can decide if the paper will be published in the journal or not.
If the paper is rejected, it could be resubmitted to a different journal with the previous review open to view.
Thanks—that’s great to hear, and I hope that model works out. We need more options to try!
That’s neat man, hadn’t thought of that. Great idea.
I like the idea of a pre-print archive in philosophy and humanities. I think it would take some power away from journal editors as gatekeepers of published work, and I think that would be a very good thing. Perhaps there should be some basic screening but why not allow a wider academic public decide on the merits of a paper instead of a couple of reviewers who may or may not give it fair reviews, or may or may not take a liking to it? Why are we so attached to the idea that journal acceptance is the only acceptable form of validation by peers for our work?
It strikes me that other disciplines (Economics) often pay their referees. In addition to the author’s suggestion of a market for reviewers, it may be interesting to simply examine successful incentive-based practices in other disciplines. How do they work? What issues do editors run into? How can we adapt these practices for our purposes? If we’re serious about reforming our peer review system, then it seems prudent to identify and study success cases (both in and out of our discipline).
What is the price of a referee report at an economics journal? I’m interested to know what fraction of my salary could be covered by the large (~50 per year) amount of refereeing I do.
Most referees are already compensated: their papers are reviewed by their peers.
If things are trickier than they once were and it feels to some people that they are reviewing many more papers than they are submitting (though this should be expected as one becomes better known), it’s likely because of the increasing size mis-match between the author pool and the referee pool: lots of grad students/very early-career researchers relative to the more established people who tend to be called on to referee.
(Of course, some of these grads/very ECRs do review work, and they are often very good at it, but by dint of their recent arrival, they are less likely to be known and called on.)
The sort of incentives paying referees might be expected to introduce:
For pay-to-publish journals, this incentivises rating highly those referees who give papers an easy ride. The more the journal publishes, the more money it makes.
For pay-to-read journals, this incentivises rating highly those referees who adopt a high-bar for novelty/interest. The more interesting the papers a journal publishes, the greater its audience and the more money it makes.
Maybe (?) that’s okay in terms of the journal landscape it produces—let a million flowers bloom—but it leads to difficulties in taking at face value any rating system for referees.
Practicalities: This is a very expensive endeavour. Paying people across the globe, quickly and reliably, is not something easily done and would certainly involve costs well beyond whatever sum goes to the referees. (I invite you to consider how easy it is to get paid by a university that is not your own employer!)
The sort of outcomes we might expect: Journals that can pay referees, well or at all, can attract more referees, achieve faster turn-around times, and generally become more attractive venues for authors. That is, big for-profit publishers who can pay will excel; smaller journals will suffer.
Another way many referees are already compensated is that they are salaried academics. The idea that we do our refereeing “pro bono” is just confused — indeed, it might be part of what encourages the bad behavior that is at issue here.
I don’t know about you John but my salary is not dependent on my reviewing anything for anyone.
If I don’t teach a class I’m supposed to teach my salary is impacted. If I stop publishing articles my salary is impacted. If I stop doing any department or college service then I’ll eventually disappoint enough of my colleagues that they’ll leave me poor evaluations and…you guessed it..my salary is impacted.
But I keep doing all that stuff and never review anything again for the rest of my life, my salary is not impacted.
*That’s* why people say that reviewing is uncompensated. It is neither necessary nor sufficient to be compensated for work as an academic philosopher. Reviewing is, in my experience, always a “good to have” but not a “needs to have” or even a “nice to have” for the job. People who review do so because they feel a duty not because they’re being paid.
If that’s why people say reviewing is uncompensated, then what they say is false. I’m paid by my university to do the work of an academic philosopher. That includes research, teaching, and department and university service, but also such things as traveling to conferences, giving my colleagues feedback on their work, answering emails from the (very occasional) person who is interested in what I’ve written, and … acting as a peer reviewer for journals. That I am not paid *extra* for these things, or that my salary isn’t contingent on my doing them, doesn’t mean that it isn’t work for which I am paid. For it’s work that clearly falls within the scope of the professional duties that my salary is compensation for.
It’s possible that your university works differently than mine and, if so, then I apologize. I’m paid to be a professional philosopher too! I agree with you that that’s what I’m paid to do but how does my university determine whether or not I’m doing the work of a professional philosopher? My department evaluates me, gives me scores for teaching, research, and service, and then my administration adjusts my future salary on the basis of these evaluations.
Reviewing for journals is, at my university, counted as “service.” Consistent with what I wrote earlier, it is entirely possible for me to get a perfect service score without reviewing for a single journal. It is not a necessary definition of being a professional philosopher, according the rules at my university and our evaluation metrics, for me to review for a single journal ever. In fact, if I tried to claim journal reviewing as the majority of my service I’d be penalized for it!
Your mileage may very well vary but the same rules have been in place at all three universities I’ve worked at. I said earlier that people who review do so out of a sense of duty and not because their salaries depend on it. My experience, in US higher education, tells me that that’s 100% right. If, at your university, your pay would drop if you ceased to review for journals then your case is, indeed, different from any I”m familiar with.
I don’t disagree that the ideal philosopher engages in excellent research, is committed to excellent education, performs service for their department and college, is conscientious of those less fortunate, is ready and willing to spend time and emotional energy on mentorship, and also reviews articles for journals without being paid to do so. But that’s the Platonic ideal and not the practical reality of doing paid professional philosophy.
The fact that some aspect of your job isn’t picked up by a performance review and translated into salary doesn’t mean it’s not part of my job.
Oxford didn’t (so far as I know, still doesn’t) do performance review at all, at least not a review that’s tied to salary. You could blow off significant chunks of your job and not be monetarily penalized (that penalty would have to be being fired, for which the threshold in practice was pretty high.) That didn’t mean those things weren’t parts of my job.
All this shows that academics have discretion in how we approach the tasks for which we are paid, that there are trade-offs between these tasks, and that some of these tasks are more essential or important (to our employers, to us, and to others) than other ones. I didn’t deny any of that. But look: if I were an accountant who did philosophy as a side hustle, and I spent an afternoon refereeing a paper, then during that time I wouldn’t be doing my job — it would be no different than if I were playing Tetris or watching TV. But I’m not an accountant: I’m a professional academic; and so while refereeing takes time away from other duties that I have (and so might reasonably be refused on those grounds), it *is* work for which I am paid by my employer.
The fact that it’s possible to get the highest score for service while doing 0 refereeing doesn’t mean that refereeing isn’t part of the way service is scored. It just means it’s possible to max out the scale with only some of the categories.
It’s possible to max out the scale of research without writing any books, which doesn’t mean books aren’t scored.
Service is such a variegated category that people do very different things within it – some do more for the university or college, some do more for the department, some do conference organizing, some do refereeing. You usually have to do *some* for the university, for the department, and for the profession, but which particular types of things in each of those three categories you do has flexibility.
Well our problem is that: (1) peer review is broken, and (2) institutional inertia and collective action disincentives makes all of the radical solutions that might fix it unlikely to ever be implemented.
How about the following for a more modest (and hence more implementable) partial solution. Journals apply a one year submission embargo (you are not allowed to submit your papers to that journal for one year) to any academics who meet the following conditions:
This system is not 100% perfect (e.g., people with very legitimate reasons for turning down a review might sometimes be unfairly penalized). But there are many good journals out there, so having a couple of temporary embargos for not completely fair reasons will not hamper your ability to get your work under review at good places. Meanwhile, this system would penalize heavily all the free riders who abuse the peer review system by turning down most review requests, or constantly missing deadlines, or submitting very unprofessional reviews. Such people would quickly find themselves with lengthy embargos at many good philosophy journals and thus struggle to place their work in good places. This would surely be a strong incentive for all the bad referees to clean up their act and the quality and efficiency of the review process would significantly improve.
Thanks for the idea. Of course I’d be happy to get paid for all of my work reviewing articles. But realistically, the money isn’t there to make it more than a pittance. I continue saying yes to review requests simply because others spend their time reviewing my articles and it’s only fair to repay the favor.
If there is one thing that Elsevier journals *aren’t* lacking, it is money. I would be very happy to referee for free for small journals, while only refereeing for Elsevier journals for a fee.
I think a more serious effort should be made to identify and engage with reviewers who are not being utilized. I hear some people say they get a million review requests a year, when I get one or two – which I complete thoroughly and quickly.
I’ve reviewed probably 5 articles for four different journals, and the editors always seem happy with my detailed work, and I always make a point of doing it within a week. But then they never ask for me to review again. No idea why.
There was a Philosopher’s Cocoon post about this a few years ago. From the discussion there it seems like many editors want to rely on a small elite with prestige credentials to do the gatekeeping work of policing who gets into top journals and so overlook otherwise accomplished scholars who lacks these prestige credentials. This is probably why you are being overlooked.
I see. They’re beggers who still insist on being choosers. C’est la vie.
I think this is a tendentious way to put a pattern of facts that can also be put in another way. Editors want to get reports from people who are familiar with the relevant literature. But there is no way to publicly make clear that you are familiar with the relevant literature, other than by publishing a paper (which any particular editor is unlikely to have read). So editors tend to fall back on names that they already know, and rely on those people to recommend alternates.
It’s been helpful that in a few journals I edit, the referee selection field now pre-populates with a bunch of names of people that some AI thinks have published relevant papers. Usually a few of them are people that I already thought of, and a few of them are clear misreadings of paper titles by the AI. But there are occasionally some helpful suggestions there that help me find people I was otherwise unaware of!
We don’t *want* to rely on people with prestige credentials (though we do want reliable people to get recognized and get those prestige credentials!)
That is some good perspective. Without wanting to be too hard on editors, since it’s plenty of work, this still seems a bit lazy.
In thirty seconds I can search PhilPapers for some people who have written on any given topic, and approach a few who have published in journals I know to be credible. It’s never been easier to ask people other than your prestige friends.
Great discussion and ideas. A few years ago I posted a series of 3 annotations to an eLife editorial by Michael Eisen, in which I made the case for an Open-ended Peer Review (OPR) system supported by peer review grants offered by the funding agencies (https://hyp.is/F8pOKAelEeqDJrOaJT9tYw/elifesciences.org/inside-elife/e9091cea/peer-review-new-initiatives-to-enhance-the-value-of-elife-s-process). For example, at an annual budget of 40 billion US$ and 1% dedicated to the OPR, NIH would be able fund forty thousand 10K grants/year for independent reviewers.
Interesting, and potentially good, idea.
One caveat: how do you get an “average rating for (your) reviews”? For example, as is my academic researcher duty, I have reviewed dozens of papers (pro bono, of course). I keep a record of my reviews, which have been thorough and time-consuming, and as of lately journals have started to formally acknowledge reviewer contributions… plus, I have the e-mail trail to account for the earlier ones… I suppose I could also dig out the responses to my reviews, which could serve as an “evaluation (or rating) of the reviewer”… But how would be such ratings standardized? And what’s more important, which authority, or organization, would be responsible for controling the quality of the reviews? Who would be giving out “ratings”?
The more I think of it, the more I conclude that the pro bono review system, by which specialized, highly knowledgeable peers evaluate the rigor and reproducibility of the research, for the sake of science, is the most appropriate. The problem is that, for some time now, novelty (whatever that means) is more important than rigor or reproducibility.
Real science needs time; and the pace at which internet generates and moves content is incompatible with that.The field is innundated with fast, cheap trash; and paying reviewers will not change this fact.
I was thinking more about giving the scientists who excel at evaluating data and observations, and have a track record, the opportunity to get funded and develop their academic career based on their skills. An Open-ended Peer Review (OPR) system funded by grants should incentivize researchers to produce and publish results and ideas that can stand immediate and vigorous scrutiny, thereby discouraging the publication of questionable or unreproducible studies.