Peer Review, Intellectual Tastes, and Some Fun
Have you ever wanted to tell off the reviewers of your manuscript?
[Dieter Roth, “My Eye is a Mouth”]
I ask you not to lower your standards, but to define your task more narrowly. You’re just trying to improve the paper, I know. But you’re doing so by inserting your own intellectual tastes. You’re trying to draw it into your own equilibrium, judge it against the backdrop of your own theoretical horizon. Instead, the only tastes that should be relevant are those prevalent in the field of research.
The author is Tom Kaspers (University of Chicago). His “Reply to the Reviewers,” was published recently in Synthese.
The piece is “a satire of the response document submitted when a paper needs revisions.” Written from the perspective of a disgruntled author responding to the comments of referees, the author ends up complaining about how some peer reviewers see their job and execute their task. The paper “defends the claim that philosophers should be allowed to charter their own course and develop theories that appease their intellectual tastes. Yet, peer reviewers often insert their own tastes when reviewing a manuscript. As a result, too many papers end up getting rejected.”
Kaspers writes to his reviewers:
I truly appreciate your tireless efforts, and your commitment to our discipline. But I worry that these efforts have been misdirected. Your most noble belief in the pursuit of knowledge and understanding, and your conviction that rigor and generality is worth every sacrifice, have plunged the system of peer review in a state of absurdity. Far too many papers presenting perfectly coherent theses get rejected. *
Having grown too accustomed to rejections, we tend to shake it off as a mere difference in intellectual sentiments—luck of the draw—and submit again, hoping to get a reviewer who happens to be more sympathetic to our style and tastes. If everyone has to submit five times to get their paper accepted, we’d need ten reviewers for every paper. We don’t have that many reviewers, so editors are forced to desk reject many of their submissions. All in all, one’s perfectly coherent theory exists in the limbo of peer review for years before it sees the light of day. And before that day arrives, there is the day of major revisions. This is the day the author, defeated and desperate, will betray their own theory by accepting every wayward suggestion the peer reviewers have made—performing an impossible and highly destructive balancing act to cater to both reviewers, even though their comments often point in very different directions.
Kaspers’ offers the following recommendations:
Instead of asking whether the thesis is true (which, I’ve argued, often amounts to asking whether you agree with it), I would like you to consider the following conditions. (1) Does it meet basic standards of clarity and correctness, e.g., does it clearly and correctly describe the theories and arguments of other philosophers? (2) Is it internally coherent? (3) Does it address well-known questions frequently asked by the field? (4) Are its answers to these questions likely to be productively original to a sufficiently large share of the readers? If you think it does, you ought to accept it, no matter how much it offends your own tastes. Not all that shall be built will appease your eye, but the philosophical land is expansive and plentiful. My hovel is but a dot on a map. And while I’d agree it might offend one’s tastes when seen in a context in which it doesn’t belong—like a Venetian palazzo on the Las Vegas Strip—in its proper surroundings, it actually looks quite charming.
You can read the whole article here. Discussion welcome.
(via Mattia Cecchinato)
Related:
How to Write Referee Reports
An Objection Does not a Rejection Make
Worst Reviewer/Editor Comments You’ve Received
The Questions a Referee Should Ask of the Paper They’re Reviewing
A Little Rough Data about Journal Refereeing in Philosophy
Advice on Refereeing Papers
What’s So Bad about Bad Philosophy?
Notably Good Experiences with Philosophy Journals
And I thought I had humor when I used Mike Tyson as a case study in a recent paper. Nice idea by Kaspers!
Finally, someone said it! Peer review feels less like evaluation and more like surviving a taste test by grumpy philosophers. Kaspers nailed it—may we all one day publish without betraying our own theories!
Yet another reason why books are a better medium for philosophy than articles.
It’s interesting that none of the recommended conditions mention anything about whether the argument is sound or plausible. Surely “internal coherence” is much too weak of a requirement?
The problem with phrasing it in this way is that there’s not really a way to separate it from just agreeing with the conclusion. For most reviewers, it would eliminate vast swathes of great philosophy.
Suppose I have in front of me a paper on modal realism by David Lewis. I don’t think his arguments are sound and definitely not plausible. If I did, I would be a modal realist. Should I recommend rejection on that basis? That’s ludicrous.
I didn’t suggest that the reviewers must agree with the paper’s conclusion. But they shouldn’t accept completely implausible arguments either.
I’m not sure I understand the point. Why would internal coherence be too weak? What makes an argument implausible if validity conditions are met? The implausibility of the premises, of course, but that doesn’t seem to be what you have in mind. If you just mean the plausibility of the conclusion itself, then why are implausible conclusions a problem?
That’s in many ways the entire point of the post, which captures quite well the intuition that many of us have that peer review often introduces arbitrary and biased conditions on “soundness” and plausibility that are often indicative of epistemic vices rather than epistemic virtues.
FWIW, I suppose I can agree that internal coherence is probably not the only argumentative norm that matters, but I also think it is definitely a straw man to foist that on Kaspers (who offered four enumerated criteria that would be relevant as argumentative norms, only one of which is internal coherence). But I just don’t understand what “plausibility” of an argument argument here means. If it just means attending to the relevant argumentative norms, then how does that cut against Kasper’s point?
Whenever my papers are rejected, it is because the reviews are morons. Whenever they are accepted, it is because they are subtle thinkers who recognize philosophical quality.
When I’m rejected, I should have known better than to submit so soon. When I’m accepted, it’s because the reviewers felt sorry for me.