An Objection Does Not A Rejection Make
“If philosophers are serious about improving the way their journals function, they need to consider not only how to improve the mechanics of the reviewing process, but also how to improve the way they criticize one another.”
What are good grounds for a journal referee suggesting a paper be rejected? Tim Crane (CEU) has some thoughts on that.
In a recent piece at the Times Literary Supplement, he discusses the low acceptance rates of philosophy journals (discussed previously here, here, and here, for example) and suggests an increase in publication:
The discipline has high standards, but the number of competent philosophers in the world and the number of articles they are trying to publish are all growing. Given this, and given the new opportunities presented by digital technology, there is no reason why the leading journals should not just publish more stuff. Of course, it might mean that publication in one of these journals may no longer be that sole decisive achievement that will get you that job or grant. But this could be beneficial: rather than evaluating someone’s work by looking at which journals they publish in, assessors would have to actually read the work itself.
However, one obstacle to publishing more philosophy, Professor Crane notes, is “the attitude of philosophers who act as peer reviewers”:
Many behave as if finding an objection to the claims of a paper is a sufficient reason to reject it, or to ask for revisions before publication. Authors are regularly asked to revise their papers to take account of a wide variety of more or less plausible objections. This inevitably results in papers that are longer than they should be, and in many cases far more boring and hard to read than the original. The whole “revise and resubmit” process also adds months to the publication cycle. In many cases, journal editors would do a service to their readers if they took a few more risks and published even those papers to which someone might—shocking as it may seem—make a good objection.
It will be difficult to improve this situation without making some fundamental changes to the way academic philosophers are trained. In the analytic tradition, philosophers are taught to write in a style that, in the memorable words of Bernard Williams, “tries to remove in advance every conceivable misunderstanding or misinterpretation or objection, including those that would occur only to the malicious or the clinically literal-minded”. It is therefore unsurprising that the criticisms often put forward in peer review can seem uncharitable, pedantic and pointless. If philosophers are serious about improving the way their journals function, they need to consider not only how to improve the mechanics of the reviewing process, but also how to improve the way they criticize one another.
The full essay is here.
Edmund Dulac, illustration from “The Princess and the Pea”
Related: “Don’t Forget to Remove the Scaffolding“
This is really excellent, Justin, and desperately needed.
I edit a journal that is pretty far from philosophy (we publish a mix of empirical analysis of data archives, experiments and surveys, economic models, history and “thought pieces”.), But I wonder whether our non-traditional approach solves some similar problems. One of our big changes has been to distinguish substance from commentary. Substance is what you do and find: how you gather, analyze and report data; how you set up and analyze an economic model; what arguments and sources you use to support historical and ‘thought piece’ claims. Commentary is everything else, including claims of novelty, contribution, importance, implications for scholars and practitioners, and matters of taste. We demand high standards for substance, but reasonable disagreement over commentary is not a bar to publication. Instead, we allow reviewers to express their own views in a discussion.
We also encourage authors and reviewers to acknowledge and embrace compromise. No study is perfect, and every method strong in some ways falls short in others. Rather than reject studies because a reviewer places different weight on strengths and weaknesses than the authors,, we encourage revisions to shore up the strengths and acknowledge the weaknesses.
So far, it’s worked reasonably well. One of the interesting results has been that we have been receiving and publishing a far more diverse set of papers, and that authors are adding a lot more substance to avoid unresolvable debates over commentary.
I’d be interested in hearing how this would apply in philosophy.
Here are two short videos on our approach.
Here is one on the role of reviewers.
Here is one on substance, commentary and compromisee.
I just watched your second video: excellent ideas and very well presented. Thanks!
Perhaps it’s just me, but I’d rather read a small number “boring and hard to read” journal articles in philosophy that address “good objections”, than a larger number of “risky” ones that overlook or even ignore them. (This follows from a more general principle which I’d apply to journal articles in physics, medicine, and so on.)
I wouldn’t. I’m much more interested in interesting and thought-provoking ideas. Philosophy is not the sort of subject in which one finds too many “correct” positions, anyway.
I take the suggestion to be: a healthy discipline would have room for both types of articles.
Further, I don’t think Crane is complaining about *comprehensiveness* per se, but rather a kind of absurd idea that no objection is not worth a response. Some objections are confused! And we *all* have to draw the line somewhere.
I take the suggestion to be: a healthy discipline would have room for both types of articles.
= = =
Yes, I think this is right. (Both that it is the suggestion and is a good idea.)
I think this is my “analytic philosophy” side speaking but I read philosophy to parse out logical issues using arguments (in my own case this takes place in the philosophy of mental illness and moral responsibility). I don’t do it to read good prose or good literature. I *want* the logical precision and attention to detail that Crane seems to bemoan.
There are obviously good and bad examples of refereeing. We’ve all also been subject to more of the latter than the former though I don’t think that the problem is the one that Crane identifies. If one of my reviewers poses a real critique an article I’ve submitted then I’m OVERJOYED! It means that they’ve carefully read my work and actually spotted a problem in it. That’s great! That’s philosophical progress. What too often happens, however, is that refereeing is such a devalued process that people do it very quickly and uncarefully and the comments display these qualities (nobody gets any real credit for it as service to the profession, it’s unpaid – arguably exploitative – labor we pay to profit-making journals, and it takes a lot of time to do well). Fixing those structural problems would, in my opinion, go much farther to improving the refereeing process than publishing more or with sitting on objections when one reviews.
There’s such a thing as being logically precise and thorough without literally anticipating and replying to every possible objection. Some of the greatest classics of our tradition would not make it through the current standards, because “you didn’t consider objection X.” And that’s when you know that you’re no longer being rigorous, but rather, ridiculous.
Right. And I’m proposing no such strawman Dan. Literally anticipating and replying to every possible objection is impossible, yet somehow more articles are published now than ever before!
But this is all that Crane is arguing. He’s not saying that sloppy work should be published, only that the ability of a referee to think of an objection should not be sufficient grounds for rejection in most cases. That sounds right to me. It beggars belief that only 4% of the papers submitted to Top 5 journals are carefully written and interesting arguments.
I just think that you were going on a stronger version of the thesis under discussion than it really is. And it *is* the case nowadays that papers are routinely rejected or returned for revise and resubmit, because some referee thought of an objection.
Daniel,
Do you think R and R is the same as reject? I guess I recommend R and R both when I think I have suggestions that will make the paper better enough to pass whatever standard I think needs to be passed for the particular venue and I think the paper is otherwise promising, and when I think I have a good reason that the paper doesn’t work at present, but there is enough there that I want to see whether there is a good enough response to what I think of as a serious problem. Those seem different to me and both seem different from an out and out reject.
For some journals it is functionally equivalent to a rejection. Overall, I’ve found that trying to satisfy all the editorial requests made in an R&R, is a poor way of devoting one’s resources. Much better simply to send the paper somewhere else.
Another top sin of reviewers is expecting an author to address everything that anyone else has ever said on the subject. Another one is expecting the author to address the referee’s own half-baked thoughts on the subject.
Agreed. Same ridiculous standard, and I’ve run into that one a lot too.
Hmm. I’m sort of torn about this. On the one hand, I agree that “there’s an objection to a premise” isn’t a reason to reject. On the other hand, I do think “there’s a really obvious and good objection to a premise” is reason to reject, unless the author has some sort of reply–in which case I think the obvious and good objection, and the reply, should be included in the paper.
I would have thought that the obvious and in some sense insolvable problem is that whether an objection is really obvious or good is vague.
“I do think “there’s a really obvious and good objection to a premise” is reason to reject, unless the author has some sort of reply.”
Few of the great philosophical works we study and do scholarship on would pass this test. Kant’s work wouldn’t. Neither would Aristotle’s. Nor would some very famous articles in 20th century analytic philosophy.
“Few of the great philosophical works we study and do scholarship on would pass this test. Kant’s work wouldn’t. Neither would Aristotle’s. Nor would some very famous articles in 20th century analytic philosophy.”
I’m not sure why I should care about this. Philosophy isn’t done in a vacuum and context really does change everything. A lot of ancient and early modern work is, by our standards, really quite sloppy and vague. It was important in its time given historical facts about who had the privilege of writing philosophy and how philosophy was written but that was long before the day of the peer-reviewed scholarly article. Its contemporary importance is owed to philosophers who have tried to make more precise these historical works to extract the real marrow hidden inside their old bones.
We don’t have to go backward to go forward. Fixing the peer-review process is important but oh my god it would be terrible if everyone started writing like Aristotle (or Wittgenstein).
It seems to me that one should care about this, because many of these figures are a hundred fold better philosophers than those working today and their work is a hundredfold more significant. That such superior work from such superior thinkers would fail to meet our current standards, suggests that the standards are out-of-whack at best.
There is no reason why one should have to anticipate and respond to every objection and there is no reason why one should have to cite every relevant source. What we are doing today is an exercise in vanity and self-importance, not anything to do with serious, interesting, thought-provoking work. If it was, we’d be in a golden age of the discipline, which clearly is not the case.
“If it was, we’d be in a golden age of the discipline, which clearly is not the case.”
Is it? Just look at what you’re doing at the Electric Agora and what others are doing at Philosophy Bytes and the Partially Examined Life and Philosopher’s Magazine and the Free Philosophy Project and the Night of Philosophy and the Ask a Philosopher both and the Intercollegiate and High School Ethics Bowls, and etc. etc. etc.
Or look at how many students we’re reaching through gen-ed classes at schools ranging from the most elite schools to open enrollment community colleges.
I’m not sure I have any clear criteria for a philosophical golden age, but it’s hard to see how this isn’t at least a contender. Is the work we do at academic conferences and in academic journals part of this? I don’t know, but the people spearheading those efforts were all at least initially part of the existing system of professional training and education.
Derek, you are absolutely right with respect to philosophy more broadly and especially its public manifestation, though even there I would suggest it doesn’t have the social/political impact it once had. But really, I was speaking purely in terms of its academic/scholarly dimension.
Cantian, were the “really obvious and good objections” to Aristotle and Kant really obvious to people at the time? If not, then their work doesn’t fail my test.
And if there are/were really obvious and good objections to Aristotle and Kant’s theories, then oh my god think how great it would have been if someone had pointed that out to them before publication. I mean, we’re talking about two of the greatest philosophical geniuses of all time. It would be phenomenal to know what their response was to these really good and obvious objections, or to know how they would have modified their theories in response to the objections. I mean, you don’t think they would have just thrown in the towel, do you?
If Kant was writing now, we might very well demand higher standards out of him. “This is great work for someone writing in 1800!” seems like a weird editorial standard to use in 2018.
Plus, Kant taught an insane amount of classes on an equally insane amount of subjects. Today, he’d have a small teaching load and TAs, and teach on fewer subjects. He’d have more time to meet those objections! And Aristotle was essentially an entire university on his own! No wonder he didn’t address every objection!
You are absolutely right, and I think the pushback you are getting is quite telling of our current era.
‘An objection does not a rejection make’. True and it does not necessarily make a ‘revise and resubmit’ either, especially because for a high-ranking journal ‘Revise and Resubmit’ is often tantamount to rejection. So if I think a paper is basically publishable but open to an objection, what I often do is this. I advise acceptance but explain the objection in my report. I then say something like this: ‘This is a point the author might like to consider in polishing up her paper, however I do NOT want to make this a condition of publication. The paper is good to go as it is.’ That leaves it up to the author and her philosophical conscience. And if she publishes the paper unchanged well that’s a possible paper for me developing the objection.
Since philosophy is a debate-based discipline it is better to combat your opponents in the open where something might be leaned from the spectacle than to stifle them in the dark.
Like Charles (I think), in my reports I usually try to distinguish between points where I think a change or reply is mandatory or necessary (Usually something like, “You argue X. But X seems to imply Z, which is false or seems very undesirable [depending on the argument here]. You need to say either why Z doesn’t follow or why it’s not a problem if it does.”) I typically don’t insist on any special sort of a reply to these remarks, but may think they are necessary. I then typically also have a lot of “suggestion” comments – places where I think the paper can be made better by tightening up an argument, including another line of thought, where a further idea should be noted, something cut out, etc. But these really are suggestions, not demands. (Unfortunately, it’s hard for me to know how these are taken, because so few journals follow the good practice of sharing results with referees. Some do – _Legal Theory_ is especially good, for example – but most don’t. That’s annoying from a refree’s perspective.)
Tim mentions the possibility of ‘leading journals’ simply publishing more papers. Fair enough in principle and perhaps in the future, but at present not all such journals have that freedom. Currently, the Australasian Journal of Philosophy (for which I am the Editor) has available a contractually mandated maximum number of pages per year, to be divided equally, or almost so, between the year’s four issues.
Ditto the BJPS – eventually it’ll all be electronic but not for a few years yet. And no, rnr ≠ reject. And yes I too generally prefer ‘interesting and thought provoking’ ideas (who doesn’t?!) but I wonder if Tim’s concern is a little overblown – certainly my impression is that when it comes to phil sci, papers are being rejected less for nit picky reasons and more on grounds of either ‘look I can drive a truck through here’ or ‘this just isn’t interesting or thought provoking *enough*’! (but then I would say something like that wouldn’t I?!)
I share your impression – but in general, online discussions of refereeing bear sufficiently little resemblance to my experience that I wonder if philosophy of science is atypical.
If these sorts of “great idea, ignoring some natural objections” articles are valuable, why not start a new journal which tries to publish only them? As a grad student, I have no idea whether such articles really are valuable; but by their fruits ye shall know them…
Props for the great Dulac. Nothing like the golden age of illustration.