Refereeing Articles That Discuss Your Work
How should you respond to requests to referee papers that are mainly about your own work?
[Paul Gauguin, “Painter of Sunflowers” (portrait of Vincent Van Gogh)]
As a mid-to-late-career philosopher, I regularly get requests to referee a paper that is substantially or even centrally about my own work. I have had a policy of declining because of the obvious conflict of interest. But I wonder if the interest journals have in finding enough qualified referees might have enough weight to override this. They will be aware of the conflict of interest and can factor it into their reading of my report. And, if done right, the assessment and, often, revision, based on response by the target author can also be a boon the quality of the journal and the eventual paper if it goes forward. I don’t think that the referee should be identified to the author, which might seem a natural suggestion (except, perhaps, once it has been accepted?). Having a paper rejected can be pretty upsetting, and without confidentiality it might be hard to report honestly.
One idea that occurs to me is to agree to referee conditional on the journal enlisting two other referees. But that obviously nullifies part of the advantage for the editor. They might still appreciate the input for quality reasons. On any variant, though, it risks contributing to insularity and stratification by concentrating power. Maybe that should settle it.
It would be especially good to hear from lead or associate journal editors (pseudonymously if need be).
Readers?
Maybe this is exclusively a concern with persuasive writing in which the outcome of a disputation is the main thing at stake? A project that settles some open questions presented by earlier work can often most helpfully be addressed by the author of that other work. Similarly for work that extends or applies a framework introduced by someone else. In most areas of academic writing, the observation that earlier work was erroneous (had a flawed methodology, an error in a proof, or an unwarranted conclusion from the data) is not typically handled within the publication apparatus, though when it is appropriate to do so can very meaningfully be evaluated by the author of the work containing the alleged error.
I think this issue is probably intertwined with the issue about whether “philosophy is about arguments” that became the focus of another recent discussion on this ‘blog. As was pointed out there, arguments abound outside of philosophy, though rarely is it considered a scholarly achievement to point out that someone else’s argument is wrong nor is it typically plausible that someone would stand their ground in the face of such observations. (Think of an advanced mathematics lecture: lots of extremely complicated *arguments*, lots of discussion about them all directed towards trying to understand or help others understand things that are not at all easy to understand, no *disputes* (as LW said, “mathematicians don’t come to blows). One can take great pride in developing this theory or constructing the arguments within it, but there is no ego-investment in the arguments per se such that others’ discussion of them could not be coolly evaluated by the person who first devised them.)
A friend of a friend once observed: in a maths talk, the idea of the question is to work out why the speaker is right; in a philosophy talk, it’s to work out why the speaker is wrong. (An exaggeration, to be sure, but it gets at something.)
I broadly smiled at this one because this mustard seed miraculously and immediately bloomed for me.
I’m wondering about the “obvious conflict of interest” here. Is the conflict supposed to be that one would have an obvious interest in accepting the paper, to increase one’s citation count, even if the paper is bad? Or that one would have an obvious interest in rejecting papers that criticize one’s work, even if the paper is good? Most philosophical papers that deal centrally with someone’s work are in large part critical of that work, so both effects are likely in play.
I’m reading this post and writing this comment while taking a break in the process of refereeing a paper that cites a few of my papers, but argues for what I think is a clearly wrong conclusion. This seems to be a case that would involve both of the “obvious conflicts of interest” that seem to arise with a paper that more centrally engages my work. I’m having trouble finding a reasonable and meaningful policy that would advocate against ever serving as a referee for papers that centrally engage one’s own work that wouldn’t also advocate against cases like this. This is probably only about a third of the refereeing that I do, so this policy wouldn’t force all my refereeing to be on topics that are more tangential to my interests and expertise, but it would mean that more of my refereeing would be. (Or else that I’m doing substantially less refereeing.)
As an editor of various levels for several journals, the hardest part of my job is finding moderately qualified referees that are willing and able to review a paper within a reasonable timespan. I think any policy that makes this harder needs to meet a very high bar to be justified. I think an editor can tell if a reviewer is treating a paper with kid gloves, or is being unnecessarily harsh just because they disagree with the thesis, and if that happens, they can call for another referee if needed. (It may not be needed if the editor can form an adequate judgment about the paper even on the basis of an obviously biased review, or if there’s already a second less-biased review that makes a strong case.)
It may be that people think that “insularity and stratification by concentrating power” is a clearly bad thing (it’s not actually obvious to me that it is, unless this concentration of power causes actual harms to someone), and that it’s worth expending a large amount of disciplinary effort and progress to eliminate it. There are many other problems of the current structure of peer review. I think we should be open to completely new models of publication, that don’t depend on anything like the current peer-reviewing process. But I haven’t yet heard of any such models that don’t cause even *more* insularity and stratification, except perhaps the computer science model, where prestigious “publication” is about being accepted to a major conference (with a written version in the proceedings) rather than being in a journal.
I’ve found it hard enough to find reviewers that eliminating the people whom the articles are about would make it near impossible. Getting “an extra reviewer” (e,g., to 3, instead of 2) is also onerous.
So I would be inclined to think that you just write the review the best you can, try to write it in the third-person (i.e., not “I didn’t say that”, but “author didn’t say that”), and so on. Ultimately, the journal editor has to sort it all out and reach some conclusion, both by reading the original paper and the reports–a good editor should be able to make sense of what’s going on.
As a sometime editor (long ago) I regularly send papers to referees who were the target of the paper. I did not consider them neutral and did not consider their evaluations like those of other referees. However, they often pointed out important misunderstandings that were easily fixed.
Similarly, I have had books of mine reviewed where the journal editor showed me the review in advance of publication and asked if I saw any glaring errors. I was not given the right to veto the review but I could point out what I thought was a simple mistake or misquote.
Both of these practices are useful, as they spare readers from needless mistakes. This is quite different from allowing authors a chance to reargue some subtle or contentious point.
I’m curious what the responses would look like if you asked junior scholars (and grad students) whether they would like the tenured faculty they’re writing about to referee their work when they send it out. It seems obvious to say “of course I’ll be as unbiased as I can as I read someone’s mediocre criticism of my work that I’ll politely suggest a rejection on or a revision so radical as to make it become my own view.”
But…in my own experience at least, the few times where it’s been *obvious* that one of the reviewer’s was someone I’ve criticized, it’s been a train wreck (to varying degrees). I understand that it’s hard to get reviewer’s but it really feels, to me at least, as if academic publishing, as it’s currently structured (and rewarded) isn’t long for this world. Too many instances of the process famously breaking down, too many personal stories of people being locked out by famous gatekeepers, and frankly too much freeriding on the part of publishers for our very real labor (this includes you, editors, who put in so much work that journals translate into $$$ for themselves).
tldr: The other half of this question is at least as interesting as the one being asked by OP and both really point to yet more mounting problems with traditional peer review.
Caligula’s Goat’s remark undlerlines some of what I meant in my earlier comment. Outside (certain styles of) philosophy it would be extremely unusual to “write about a tenured faculty member” or to write about the work of a senior researcher in a way that could be mis-phrased as writing about the person. The ego-investment in arguments and stances and positions is a peculiar feature of a style of academic philosophy and leads to peculiar problems.
I suppose a junior scholar writing about black body radiation, the fig wasp life cycle, efficient sort procedures, or properties of the Conway knot would prefer that the scholars who first observed, defined, or provided the theoretical framework for predicting these things to be involved in assessing their work, both before and during the peer review process, than not.
I would suggest that “ego investment” in arguments and positions is not particularly unusual in the social sciences. Nor is it extremely unusual to find, say, a political science journal publishing a note or an article (sometimes with a reply) that criticizes the arguments and/or conclusions of a recently published article (and necessarily those of its author). It may not happen frequently, but it certainly happens, or at least did when I was following pol sci and Intl Relations journals a bit more closely than I do now.
We could only be so lucky to have ego investment in one’s published positions b unique to philosophy. Off the top of my head, law, economics, psychology, linguistics, English lit, and sociology all involve significant “ego investment” in one’s views. To take a natural science example, I think it’s naive to think that people like Wilson and Coyne have no ego stake in whether group selection is the source of cooperation.
It is hard to get referees, but it shouldn’t be. Here is a rough guide to how often you should agree to referee.
On average every article is refereed 3 times before it is accepted or retired to the bottom drawer. Each articles has 2 referees per submission. Thus, it takes 3 x 2 referees per publication and the same number for papers that go unpublished. There are about 7 articles ultimate rejected to every one published. Thus, on average there are 3 x 2 x 8 = 48 referees called upon for each article submitted (whether ultimately published or not). These numbers are rough guesses and I’m sure others can do better. If only a quarter of this is a realistic number, it still gives some idea of what we each ought to pay back. Be willing to referee a dozen articles for each you publish.
Surely this double-counts? If every article is on average refereed 3 times before it’s accepted or retired, and each refereeing involves 2 referees, that’s 6 referees per article submitted. The acceptance/rejection ratio doesn’t affect the number of referees per article submitted.
Jim’s formula is for “each article you publish”.
But the post says: “Thus, on average there are 3 x 2 x 8 = 48 referees called upon for each article submitted (whether ultimately published or not).“
David is right. I misspoke. I should have said 48 referees for each article published.
Most articles are desk rejected. I’d bet the average number of referees per submitted paper is quite lower than 2.
Good point. I had forgotten that when I was an editor, I rejected about half the papers without sending them out to be refereed. That’s one referee instead of two. So, cut the total number given earlier in half to 24. Still a lot.
I find the entire process of reviewing work which discusses my own work both fascinating and worrying – often, I read a sentence like ‘Jones says p’ and think, ‘ridiculous- I never said that!’ Then I look back at my own published work and realise I did say it, or said something that could be construed that way (ironically, sometimes as a way of responding to a referee’s comments). This may not have much obvious relevance to the question but I think it does – review only if you can treat your own work as something you happen to know quite well, but from which you have distance
I agree: it seems to me that a major worry in reviewing discussions of one’s own work is that the original author is often *not* in the best position to evaluate whether a critic is fairly engaging with what was written on the page.
I would characterize at least one referee report that I’ve received in this manner: the philosopher that I criticized had written P (this was not up for debate) but insisted that he really meant Q (and that this was obvious). Perhaps he was right, but this seems like a case in which the original author’s judgment is likely to biased in ways that we might want to avoid.
(Of course, this also seems like a problem that it is relatively easy for editors and referees to be sensitive to, and not sufficient to motivate refraining from refereeing except in very specific cases.)
I’ve got a co-authored paper which has the following structure “[first half] Here’s an objection to X; [second half] now you might respond by saying Y, but then there’s a further problem Z”. And there’s a published response to this paper which responds to our argument against X by saying Y. The response doesn’t mention that the entire 2nd half of the original paper anticpates response Y, nor does it discuss problem Z. Neither I nor any of my coauthors were given the opportunity to referee this paper, and it seems pretty clear to us that whomever did referee it hadn’t read the original paper, at least not recently, and were just going off the highly incomplete description of our argument that was given in the response. Point is: I’d rather have someone who’s *familiar* with the arguments refereeing the paper than someone who’s not.
A colleague recently refereed a paper that was on her work. The paper objectively misinterpreted her argument, but since she considered it a reasonable enough misunderstanding she let it slide with only a minor note about an alternative possible interpretation, which she didn’t insist upon. If we could trust everyone to be so unbiased I expect these kinds of concerns would not arise. (The authors in question chose not to consider the alternative interpretation… so perhaps the point should be that we just can’t trust people after all…)
To add to that: I’ve also had papers referees who insist that “presumably” this is what the target of the discussion “really meant”. In both cases, I know the targets personally, and they’ve told me why it would be absurd to think that this is what they “really meant”. Again — I’d rather have someone who’s familiar with the work.
Parallel conflicts arise when refereeing papers that are in your area of expertise but do not discuss your work. You have probably written something that is relevant, and you are probably predisposed either to dislike the paper for not engaging important literature (viz., your contribution) or to demand that it be revised so as to do so.
If such conflicts mean you shouldn’t referee, then you shouldn’t referee any paper in your area of expertise.