Pay Referees Per Mistake Caught?
James Stacey Taylor, a professor of philosophy at The College of New Jersey, is concerned about the problem that “scholars are not verifying the accuracy of their sources,” and offers up a solution.
Writing at Times Higher Education, Professor Taylor sees this problem as a rational upshot of the incentive structure of academia:
[C]hecking sources is time-consuming, and if a claim is entrenched then it might be reasonably assumed that others have verified its accuracy. Moreover, checking sources is unlikely to yield much professional benefit. If you discover that the sources cited do support the claim that was made, then you haven’t advanced your research by checking them. If they don’t – well, academic journals have little interest in publishing corrections of exegetical falsehoods. And even if you are able to publish your discovery – perhaps in a publication that merely documents the errors of others – you may earn yourself a reputation as a pedant, rather than as someone who advances the field. So neither authors nor referees will receive external benefit from the time-consuming task of fact-checking.
The solution, then, is to introduce new incentives. Here’s his version of that: “pay referees bounties when they detect errors.” Where’s the money coming from? He says:
these bounties should be paid by the authors in whose manuscripts they were detected – and they should be paid whether or not the manuscripts are accepted for publication. This would provide authors with an incentive to avoid error in the first place.
There would be different payouts for different kinds of errors:
Referees should thus be paid different amounts for detecting different types of error: small bounties for detecting an erroneous bibliographic entry, larger ones for identifying misquotations, with the largest of all being reserved for identifying misrepresentations of primary sources.
He doesn’t go into the details of how this arrangement would operate (perhaps escrow accounts that hold the functional equivalent of security deposits?) but presumably there is some way to work that out. Still, there are other questions such a proposal raises, especially for DN’s readership:
- Are problems with citations and the accurate representation of source material a problem in philosophy?
- One problem noted often in philosophy is that relevant sources are overlooked (for example, here, here, and here). Should identifying such overlooked sources also earn a referee a “bounty”?
- There’s a sense in which a good chunk of philosophy, especially the history of philosophy, is already involved in a kind of identification of “misrepresentations of primary sources.” Presumably that work is not what Taylor aims to incentivize with his proposal, but then what would be a workable conception of “misrepresentation” for philosophy?
- To what extent do proposals like this represent a shift in the general assumptions we make about scholars and their work? One response to recent academic “hoaxes” involving fake data was that they don’t show much beyond the fact that the academic peer review system operates on assumptions of good faith, and so can be tricked by malicious actors; and since such malicious actors are rare, we don’t need to try to restructure the system with even more burdensome forms of review. Have we become more cynical?
Discussion welcome.
Suppose I claim that a passage from a paper constitutes an example of someone holding some position or other.
If the passage doesn’t explicitly assert the position, but what is explicitly asserted very straightforwardly entails the position, is this an exegetical falsehood? If the passage doesn’t explicitly assert the position but, when considered in tandem with its author’s other explicitly asserted views, is most naturally understood as committing to the view, is this an exegetical falsehood?
I take it that the answers to questions like these are “maybe, I don’t know, it depends a lot on the context”. If so, then such cases of ‘exegetical falsehoods’ just don’t seem like the right kind of thing to be expecting editors (?) to be adjudicating on, especially if money is changing hands!
Does philosophy contain lots of cases of purported exegetical falsehoods that aren’t more-or-less like the above cases? My hunch is not, but then I suppose Taylor’s response will be “that’s because no-one’s motivated to look for them!”
The most common error among the papers I review in this regard is the citation of empirical work that has already failed replication or is obviously p hacked. I always point it out.
There probably is some degree of straightforward misrepresentation in philosophy. But more often I would bet that what we find is interpretation or attempts at charity. Differentiating between the two is a challenge, and charging someone on the basis of a different interpretation of the text seems pretty absurd to me. So there would have to be some kind of adjudicative process, which I’m sure no one wants to do. This also probably creates some bad incentives for referees. It’ll increase nit-picking in a way that I doubt would improve the quality of philosophical work that gets published.
I agree with all of this! My suggestion was aimed at reducing straightforward errors, such as “A wrote P and P is wrong” when A wrote “not-P”, or didn’t mention P at all, or “some might hold P, but P1 is the correct view”. I’m not sure how often this sort of misrepresentation occurs in philosophy (and I suspect that when it does it is usually inadvertant) and I suspect it occurs fewer times than in other fields (e.g. history) simply in virtue of the nature of the discipline. But if people don’t check sources the claim “A wrote P” might enter the literature, and this could lead to A’s actual views being unfairly dismissed.
I wonder if there is room to be inspired by how the lawyers do things? I know that some graduate students end up getting gigs managing journals. I wonder if more graduate programs could start journals, where the majority of the managerial work is done by the faculty (including finding referees), but where graduate students are engaged in the close reading and cite checking?
I wonder if graduate departments could convince universities that this was not an unreasonable use of research money. (Think of all of the graduate students in other fields who do not teach.) And I wonder if this could be beneficial to graduate students, especially if the journals are field specific, as they get to work through cutting edge work and get to network a little bit (and get out of teaching). And I wonder if this would help us develop more open-access journals hosted by departments.
I would happily submit the UCSD Philosophical Journal (Fight on, with mighty Triton spirit!)—as well as to the Yale Philosophical Review and the Wisconsin Ethics Reader…
(I am sure there are endless downsides and impossibilities. Still, it is often worth kicking tires!)
One problem I’ve heard people complain about in law reviews is that, since the refereeing work is mostly done by students, it mostly consists in checking footnotes, and not evaluating whether the arguments themselves are good or interesting or novel or worthwhile. If we can assume a source of money that is sufficient to do this, then paying grad students for the reference-checking while still relying on experts for the main refereeing would be a nice idea.
But how many grad-student-hours are needed to check the footnotes in one article? Assuming this can be done in two hours, we’d need to collect enough for each journal submission to pay for two hours of grad student labor. (I don’t know if $30 at $15 per hour is enough, or if you need to pay for health insurance and other benefits, or whether grad students get a better wage than $15 an hour.)
When I was on journal, editors (also students, but advanced students) were responsible for an initial analysis of suitability for publication/the journal and a pre-emption check (is this an original contribution to the literature). If a piece was accepted, an articles editor would then work with the author closely, making suggestions for substantive improvement of the piece and some basic editing. Only after that would it go to a source cite, where first year students would work on formatting any improperly formatted citations and double checking that all the cited claims match the sources cited. Pre-emption checks aren’t always fully successful when the senior students are typically second or third year students, and close attention to the literature writ large isn’t central to their education or career goals, but overall I think the process is actually a wonderful improvement in terms of both seeing a piece grow during the publication process and with respect to reducing errors – and issues with pre-emption checks in the law literature might be naturally mitigated by differences in the student body.
Where does the money come from that pays the bounties?
“Importantly, these bounties should be paid by the authors in whose manuscripts they were detected” Because, you know, incentives
Law reviews scrupulously check citations and factual assertions in the articles and essays they publish. Maybe philosophy journals should do this, too. But paying bounties to peer reviewers would be the wrong way to go about it. It would not provide a guarantee to reviewers that time spent fact-checking would be compensated. It would not provide a guarantee to readers that the articles they read have been fact-checked.
If we want philosophy journal articles to be fact-checked in the way law review articles are, journals should hire people to do the job.
Even when keeping my opinions about economists and anyone else who writes about “incentives” very deep inside of me, I am baffled by this.
When I say that Frege has often been misunderstood, does the contrary majority scholarly opinion make this a mistake? When I claim that Schlick said something about “evidence”, while the locution he used was actually “truth”, is that a mistake?
How in the everloving … does one want to “fact-check” for understandings and interpretations? Is Professor Taylor perhaps a verificationist who thinks that meaning only exists in the fact-checkable?
I’m not suggesting that one fact-check different interpretations of primary sources–that would clearly be a category mistake. My suggestion is much simpler: That if Smith writes that Jones claims P, while Jones claims not-P, that be flagged as an error. Or if Smith writes that Jones claims P, when Jones writes nothing of the sort, that be flagged as an error.
I believe the point being made in the previous comment is that in philosophy we’re often faced with uncertainties regarding exactly what Smith and Jones are saying in the first place.
I believe the point being made in Taylor’s comment is that in philosophy those uncertainties can be left untouched and there would still be many other cases still worth fact-checking
Absolutely! 🙂
In fact, I recently came across a case where Jones himself was mistaken about what he had written 10 years ago. These things happen, especially when trying to account for Smith’s views using Jones’s framework.
Isn’t the broader problem with this that it makes attempting to publish massively more risky for adjuncts and other minimally paid/supported folks.
An excellent idea. We should also pay police per arrest, pay teachers per student graduated, pay fishermen per piece of fish sold, and pay university compliance officers per proceeding instigated against a faculty member.