Referees With Attitude Problems


Everyone involved in the academic journal publishing process, it seems, is overworked. It’s true of the editors, of course, but also of the referees who say yes. And when people are overworked, they often become especially concerned with how their time is used up, by themselves and others, and frustrated when they feel their time is wasted.

slingshot pencil pic

When you have to referee a paper that you think is especially bad, there’s often that sense that your  time is being wasted. The thought is that the author should have known better, or the editor should have known better—that someone, anyone should have known better and stopped this paper at some point before it became your responsibility to deal with. And so you get frustrated. And sometimes, you take this frustration out in your referee report. This is one reason referees sometimes act like jerks. (There is, of course, the phenomenon of refereeing “like a jerk except you are an actual jerk.”)

Sometimes the jerks are self-aware, and wrap up their jerkiness in the guise of helpful (but condescending) assistance, or disquisitions on the nature of philosophy or the philosophy profession. The result is referee reports like the following, which a reader sent in:

In order to get their work published, scholars feel they have to convince us that they have something new to say. This feeling is justified, but it puts pressures on the authors of articles that sometimes make what they write counterproductive to their own end. They are so insistent on the originality of what they write that it only makes a reader notice how little new there really is in it. The more insistent the author is about this, and the more relentlessly the paper attacks the existing interpretations, the more a reader becomes aware of how little there is in the paper that is new. We see only the author’s exaggeration of the importance of new turns of phrase and new ways of saying the same old things. This can be done even in papers that are impressively detailed in their analysis, their footnotes, their formalizations. The paper shows impressive erudition and scholarship, and philosophical skill. But the author knows this won’t be enough unless the reader is convinced that everything written previously on the topic has been hopelessly wrong, and that the world was just waiting (though it was not aware of the fact) for this author to come along and get everything right at last…

[some criticisms]…

The next phase of the paper makes a number of points about [redacted]. These are not new, and they are not presented with any special perspicuity that I could detect. If Author is saying anything original in these pages, I missed it. At this point I begin to be bored with the paper. But I also imagine Author, when reading these comments, as reacting with fury at my being so obtuse as not to appreciate the wonderful originality of his presentation of these matters. This makes me glad I am not in the same room with Author, and there is no china handy that he can throw at me.

This is hardly the worst example of the type—feel free to share your examples in the comments—but it illustrates the point well enough.

The point is not that the referee is mistaken. I haven’t read the paper, so I wouldn’t know. It’s just that we’d probably be better off without the bad attitude. If you don’t think the paper should have been sent to referees, instead of barely concealing that in your comments to the author, tell the editor. Save your view of the profession for a blog (I accept guest posts, you know). And save time; remember this advice:

The main job of the referee is not:
1)  To help write the paper as a quasi‐coauthor
2)  Make an unpublishable paper publishable by directing the research
3)  To ensure that the paper cites the referee’s work

To which we can add:

4) To be a jerk.

If you follow these guidelines, next time you get a frustratingly bad paper to referee you have at least one reason to be happy: it shouldn’t take long at all to move it from “to do” to “done.”

Horizons Sustainable Financial Services
Subscribe
Notify of
guest

32 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
E
E
7 years ago

I’m not sure the following complaint is totally on topic, but this seems to be as good a place as any to voice it:

It drives me nuts when referees recommend rejection on the basis of the claim that the argument under discussion in a paper doesn’t merit engagement, even though the argument was published by Oxford University Press, the Journal of Philosophy, or some other supremely prestigious venue, and even though the argument has yet to receive any critical treatment in publication.

I understand the prudential reasons that editors have to let such rejections slide. But when OUP, etc. publishes something, then it seems safe to say that at least some hard-working people take the published material seriously. Why isn’t that enough to scratch ‘argument doesn’t merit engagement’ from the list of reasons for the referee to reject?

Scott Forschler
Scott Forschler
Reply to  E
7 years ago

I’ve actually twice submitted papers critical of an article published in the same journal, which were both rejected even though neither reviewer identified any flaw in my criticism, pleading only that wholly negative articles must be subject to a “higher” standard than original research. Indeed, they both admitted that the original article was indeed flawed, one even suggesting that my piece was cheaply hitting low-hanging fruit, i.e., the original article. I found it odd that the original argument was considered good enough, while identification of its flaws (however glaring and obvious) is not!

Instructor Gadget
Instructor Gadget
7 years ago

Personally, I’d prefer a mean report like the one in the OP to what I got yesterday.
“COMMENTS TO THE AUTHOR:
Reviewer #1: This submission is not sufficiently clear or well argued for publication in [redacted].”
Ouch! LOL!

GradStu
GradStu
Reply to  Instructor Gadget
7 years ago

I have received that report almost verbatim! Someone must just have a form.

Steven French
Steven French
7 years ago

I know we’ve banged on before about the BJPS blog (‘Auxiliary Hypotheses’) but you can also find some guidelines about being a referee there: http://thebjps.typepad.com/my-blog/2015/03/howtoreferee3.html

Doc F Emeritus
Doc F Emeritus
7 years ago

Wonderful article! The jerkiness, if you will, of reviewers is not due to overwork. Seriously, reading bad articles results in overwork??? try working 9 hours a day in a factory and then going to grad school at night — that will wear you out! But reading articles, good, bad or indifferent? Not “being overworked”.

The attitudes of some [and please note the word SOME (not more than a minority, and a small minority of reviewers)], reviewers is surely due to their own expanded egos, and feelings of self-importance. But, bottom-line, there is no need to be rude, denigrating and making comments as if you were God-on-high. Helpful, sensitive, and polite comments are possible. But it is much easier to be a jerk — takes less thought and creativity.

I, again, note that this is a way of acting for a tiny minority of reviewers (in my experience). But for the rest, there is no excuse to elevate yourselves to the status of saying that if someone writes a bad paper you have the right to denigrate them. Don’t sign up to be a reviewer if you believe bad papers submitted are a waste of your time to read. If the authors “ought to know their papers are bad, and should not submit them”, then what use are you?? Self-editing and reviewing would replace the need for reviewers. Just be decent in your reviews and we are all better off.

Charles Pigden
Charles Pigden
Reply to  Doc F Emeritus
7 years ago

‘If the authors “ought to know their papers are bad, and should not submit them”, then what use are you [the referee]?? ‘ This is like asking ‘If criminals “ought to know their actions are bad, and should not commit them”, then what use are you [the police officer]??’ It is sometimes the duty of a referee to explain to an author exactly why their submissions are worthless, in the hopes that they will either submit papers that are less worthless in future or that they will stop submitting papers altogether.

Ed
Ed
7 years ago

These people are of course so nice and friendly otherwise, reflecting the great professional climate in the field. It is just when they are refereeing and overworked that they become jerks. Yeah, right.

Doc F Emeritus
Doc F Emeritus
Reply to  Ed
7 years ago

There is something in what you say. Very nice people can get nasty in certain contexts. The again, historically, nasty, negative reviews have been venerated (such as Dorothy Parker, Pauline Kael, and others). It’s what we need to put up with and put into perspective. yeah, ain’t pleasant, but part of the field.

Doc Forsberg Emeritus

A casual stroll through the lunatic asylum proves that faith does not mean anything.

DefNotUsingRealName
DefNotUsingRealName
7 years ago

Last year, I went back to an older paper that I had been urged to revise and send out, which I did. One of the two referee reports I got back was the following:

“Unfortunately I can’t recommend this piece for publication. There are too many inaccuracies, areas where clarity is lacking, and mischaracterisations of key views for the central argument to really come out (that despite the fact that I think there is an interesting argument to be made in the vicinity).

For example (from page 1 and 2 alone):

1. The citations and descriptions of [blank] and [blank] in footnotes 4 and 5 are wrong.

2. The key claim contains turns of phrase which are neither helpful nor clarified.

3. The characterisation of arguments for denying [blank] on p.1/p.2 isn’t right in a lot of different ways.

5. Footnote 6 is v. peculiar.

There are many more examples.”

Ouch. I let that paper die from old age.

ejrd
ejrd
Reply to  DefNotUsingRealName
7 years ago

Def, is this the ENTIRE report? If so, I agree with you that it’s a poor one though not really jerky or mean, just extremely vague and unhelpful. However, 1-3 seem like good reasons, to me, of rejecting a paper. If the reviewer explained what they meant by 1-3 then I wouldn’t think this was a bad review, actually. Point 5 is obviously not super helpful (unless fleshed out).

Doc F Emeritus
Doc F Emeritus
Reply to  DefNotUsingRealName
7 years ago

Not sure what to say except that the comments are not personally offensive and seem for the most part professional. I get the hurt one feels when rejected (boy, I have had papers trashed, too, but if done within professional bounds we gotta live with it, right?). I see nothing in the review that are outside professional standards.

Doc Forsberg Emeritus

A casual stroll through the lunatic asylum proves that faith does not mean anything.

DefNotUsingRealName
DefNotUsingRealName
Reply to  Doc F Emeritus
7 years ago

Yes, ejrd, this was the entire review. And the other reviewer was a helpful reject/R&R sort of review. Notice that I said I let the paper die — so I didn’t necessarily think it was a bad review; I mostly shared it for the ‘ouch’ factor. But I agree with ejrd that comments like that without explanation aren’t real helpful, especially since I think 1 and 3 aren’t obviously true (I have published on both of them before, in good journals) — they could be true, but I wonder the ref’s evidence.

Note also, that one can give a pretty brutal report in kinder and less kind ways. Perhaps it’s over and beyond, but I always try to tell authors what I think they did well. The other reviewer for this paper did that, which makes the review process psychologically easier.

ejrd
ejrd
Reply to  DefNotUsingRealName
7 years ago

I completely agree with this. I was lucky, early on, to receive some really great referee reports that I used a model for my own. I also think I was lucky to come from an interdisciplinary background and so knew that philosophers were encouraged to be especially gruff (some might say toxic) when offering feedback.

JBR
JBR
7 years ago

In my limited experience, I do not think I have ever received a “jerk” referee in any of the fields I have published in (phil, psych, and neuro). I suspect part of this is luck to be sure. But I also spend a lot of time trying to write papers in a way that I think a referee will *like*; that is, the sort of paper I would want to read, if I was a referee (e.g. intro is very clear about the source of import and originality of the manuscript, the structure of the paper is transparent and easy to follow, and no digressions or needless detail). I think this can make the difference between, for example, whether a reviewer brings up a serious problem with the paper and demands rejection, or brings up the very same problem, offers a plausible way to address it, and suggests an RR.

In contrast, I worry I am a “jerk” reviewer. I don’t think I have ever reviewed a phil paper where I didn’t think the author(s) could have made the paper much much more *likeable* for the referee. So I am usually very critical of both the style and substance (though I always include genuine comments about how to improve the paper); often I offer a version of “the author should put more work into editing their manuscript before submitting somewhere else” (ouch!). In this respect, I sympathise with the “jerk” quoted in the post. Based on their comments, the author should probably have done more to make their paper “likeable”.

Pendaran Roberts
Pendaran Roberts
7 years ago

I’ve had everything from extremely helpful comments, to jerk comments, to even insane comments. For example, I had an R&R rejected once because the referee said I was too eager to revise the manuscript in line with his comments. What?

Jerk
Jerk
7 years ago

Is the following a jerk thing to do: I recently rejected a paper after reading only 7 pages. I typically assume an author deserves more, but it was clear after 3 pages that the paper was unpublishable. What would you do?

just a thought
just a thought
Reply to  Jerk
7 years ago

If the discussion/argument in the first 7 pages was so unclear that it left you very confused, then — since presumably what follows in the paper will somehow depend on those 7 pages — it’s reasonable for you to assume that you’re not going to understand the rest of the paper. If this is the case, then maybe it’s not too bad to stop reading.

ejrd
ejrd
Reply to  Jerk
7 years ago

Did you only read it once? It’s possible that the problem was with you and your reading conditions (headache, tired, etc.,). I think reviewers have an obligation to at least make a good faith effort to read an entire paper before deciding on rejecting it. I hope you don’t do the same with your students papers when you grade them.

M
M
Reply to  ejrd
7 years ago

I don’t do that with my students’ papers, but working with a student on a paper is a very, very different task than refereeing a paper.

Perplexed Marketeer
Perplexed Marketeer
7 years ago

I wonder whether this is a jerk referee comment:

“The paper relies heavily on an account provided by [redacted]. This account hasn’t really been taken up substantively in the literature, so it seems to me ill-advised for the author to rely on it in this paper.”

I wonder how many other papers have been given the same criticism, resulting in a self-fulfilling objection. Doesn’t an account get “taken up substantively in the literature” by accumulated instances of people making use of it? And if the first instances (it’s not like the account is that old–five years or so, and belonging to someone who has continually published in the field since) are rejected because no one else has cited it, that will doubtlessly give the impression that no one finds anything positive about it.

Eric
Eric
Reply to  Perplexed Marketeer
7 years ago

You’d have to say more about what you mean by “rely heavily on.” I could imagine, for example, that it meant this: You are criticizing Xism, and you rely on the fact that Q characterizes Xism as including commitment P, and then you go on to criticize P, and conclude that Xism is flawed. In a case like that, the paper would depend, to a large degree, on Q’s account of Xism being widely regarded in the literature as accurate.

Perplexed Marketeer
Perplexed Marketeer
Reply to  Eric
7 years ago

In the paper I criticize philosopher X’s account. Philosopher X provides a conceptual framework within which his account is then presented. I suggest that philosopher Y’s expanded framework lends more clarity to the central issue and use it to frame my criticism. I give philosopher Y’s reasons and my own argument that these reasons are compelling for adopting the new framework. I would have been fine with a reviewer giving some justification for the claim that the expanded framework is not necessary or implausible on its own merit. The only reason given, though, that the framework should not be implemented was that it had not been discussed enough in the literature. The review was, of course, supposed to be anonymous–I have doubts that that had happened (see the comment of AnotherSomeoneorOther below for an idea of why I had doubts)–but philosopher Y was, of course, cited in the paper. I wonder whether the reviewer just didn’t think philosopher Y was popular enough to justify my reliance on his framework.

AnotherSomeoneorOther
AnotherSomeoneorOther
7 years ago

As others have said, I think it is unfortunately not (just) that referees are overworked that explains the occasional jerk referee, but simply that they are anonymous that can really bring out the jerks in some. I’ve been shouted down during a talk, why would I expect more civil behavior when my interlocutors are behind a veil of anonymity?

Having said that, most of my referee reports have been non-jerky or even extremely generous.

On a different point, another way to be a jerk referee is to be a googling referee. We all have google analytics on our webpages, and it’s built in to academia.edu sties now. If you are an obvious expert on some topic and are from some little college town with one philosophy department in the vicinity, and you poke around my webpage to decide whether or not I’m worth a real referee report, I will be able to see in analytics that you did that. Of course, I won’t be sure it was google refereeing, but I can guess. Please stop this. It is incredibly unethical.

Professor X*
Professor X*
7 years ago

I was lucky enough to get published the first four papers I ever submitted. The journals were not top journals but solid middle ranking ethics and political philosophy journals. Feeling pretty pleased with myself I submitted my next paper to a top political philosophy journal and received a smack down that has had a lasting impact upon me. The report was a one-liner: ‘This paper is totally devoid of original thought.”

Professor Y*
Professor Y*
Reply to  Professor X*
7 years ago

Professor X*, which journal was this? Your response is anonymous, and I think it is important to share such experiences.

FWIW, I had a very similar experience with Philosophy & Public Affairs. I can’t go into the details without jeopardising my anonymity, but whatever else one would say about this paper, it did contain original work, and I personally believe it’s one of the best pieces I ever wrote (and I’ve published a lot in the “solid middle ground” and a bit in the top). Sure, I know that we’re the least objective to judge the quality of our own papers. Yet my colleagues were equally baffled. My hypothesis is that either the editors didn’t read the paper (for whatever reason), or else they systematically reject papers that do not stem from a prestigious university or are “sponsored” by a famous philosopher (I’m working outside the Anglophone center of the academic world and didn’t get my PhD from Harvard or Oxford). Triple blind reviewing is the only solution to such nepotism. My paper is now forthcoming in another journal, after having received both helpful and very nice referee reports.

So it’s not just referees that are sometimes jerks: (associate) editors can be too.

LostInTranslation
LostInTranslation
7 years ago

Many years ago, after a particularly unpleasant referee’s report, a friend of mine (who was a few years ahead of me) pointed out that by the time a referee is pointing out that your paper is useless, unclear and full of offensive non-american spellings, you’ve probably done something to annoy them that has nothing to do with using a ‘u’ when you spell ‘colour’. I didn’t understand this mechanism at the time, but with experience I have come to think he is right.
Whilst I think there are a number of ways you can offend a referee (such as simply not writing in a style consistent with the referee’s taste), one of them would seem to be endorsing a premise or a conclusion that the referee ‘simply can’t agree with’. Of course, arguing badly, making faulty inferences, is a good reason to reject a paper. What is not a good reason to reject a paper is that the referee doesn’t like something you are assuming or the conclusion that your otherwise good argument ends you up with.
What has really surprised me, though, has been the number of people I have recently spoken to about this who have said ‘yeah, I used to be that guy’. These were otherwise decent seeming people who said that it took them years (of snarky) referee’s reports to realise that not liking where the author’s argument ended up – with all of the steps being good along the way – or not liking one of the author’s premises was not reason enough to a. reject a paper and b. be a complete tool about it.
Why this is something that people had to learn from experience is a little but puzzling to me, and it makes me think there is something of a broader problem with people not understanding what their role as a referee actually is (and being the gate-keeper of your own sacred ideas is not one of them!).

JT
JT
Reply to  LostInTranslation
7 years ago

This brings to mind the guest post on ‘intuitive bedrock’ (or something like that–too lazy to look it up) from a few weeks ago. I’m still not sure if the agnosticism that it ultimately recommended is right, but it does seem that we should adopt an (weaker?) attitude to that effect much more than we do. If anything, it’ll save our tables from being thumped so vigorously and constantly.

Ed
Ed
7 years ago

Here’s another thing. We philosophers love to hear our own voices, so we write and write when a short rejection pointing out the main concern would do.
I recently reviewed a paper that was quite bad, for a psychology journal. There were other reports as well. Another one from a philosopher, which was even lengthier than mine, and a short paragraph from a psychologist. We all rejected, so that wasn’t the issue. It is just the we philosophers tried to demolish every argument in the paper. Sometimes that’s helpful, in other cases it is either posturing or lack of confidence on the part of the reviewer. Being “thorough” is not the same as being a jerk, but I think the two often do go hand in hand.
I’ve also seen referee reports in other sciences; seldom as long winded as in philosophy. In my own experience even HISTORIANS were better reviewers…

Steven French
Steven French
7 years ago

Is this a ‘jerk’ comment: ‘ the paper reads like it was written in five minutes’??!

In response I shall of course endeavour to cultivate a writing style that at least makes to appear as if the paper was written over a period of several months, tortuously, with much angst …

Scott Forschler
Scott Forschler
7 years ago

How often has anyone received a report which egregiously misrepresented the content of your submission? I don’t mean insisting upon a different view of a contentious issue, like rejecting your argument that realism and constructivism are compatible because the referee didn’t think they are. I mean, saying you didn’t cite anyone in support of claim X when, in fact, you did; saying that you were assuming Y when you explicitly rejected Y, or saying that you ignored the relevant point Z when, in fact, Z is an explicit part of your argument.

Scott Forschler
Scott Forschler
7 years ago

Part of the problem is doubtless that reviewers are overloaded with work, and philosophers under too much pressure to publish. Does anyone know of any good way to volunteer for reviewing duties other than just writing the journals one at a time? Shockingly, I have some free time I could use for this, and keep hearing editors complain about their backlogs which I’d be willing to help with….