Deciding which Papers to Referee


Some philosophers receive an excessive number of requests to referee papers. How should they go about deciding which papers to agree to referee?

Of course the paper should be in one’s area, but even that criteria leaves some people with more requests than they could reasonably be expected to fulfill, and so, with a decision about which requests to accept. One might use a random method, such as “first request of the week / month / quarter, I accept, the rest I decline” or something less systematic, like, “I don’t feel too too busy at the moment, so yes I’ll agree to referee this one.” Or one could be more strategic and make decisions in a way aimed at affecting the profession. Along these lines, one philosopher writes:

Not infrequently, I receive requests to referee papers which don’t cite any papers by women. In many such cases, it is easy for me to think of a paper by a woman philosopher squarely on topic quickly off the top of my head. As a published author, I think I am obligated to referee some papers each year.  Since I can’t accept all referees requests, my policy is to focus on those papers on topics about which I am among a relatively small number of experts.  But even with that restriction, I still receive more requests to referee than I am obliged to accept on grounds of fairness.  (I receive 1 or 2 such requests each week.) 

I have very few options at my disposal for helping the profession change its pattern of under citing women. Deciding which papers to referee is among the few tools for influencing the profession in a positive way that I have.  Putting all of these considerations together, I am sorely tempted to adopt the policy of not refereeing any papers that could easily cite papers by women, given the topic, but don’t. 

I would be interested in hearing others’ thoughts on this, to help me think about it. I would be interested to know of any good moral objections to such a policy.  I’d also be interested to hear whether others who don’t find it objectionable would consider adopting this policy. I wonder whether, if it were known that a number of us have adopted such a policy, we might together make some difference.

The idea, I take it, is that authors would come to know that it will be less difficult for editors to find referees for their papers if their papers cite relevant works by women, and that if they fail to do so there is chance that it will take longer for their papers to make it through the review process, if they get through at all. Of course, such a strategy could be employed on behalf of other groups whose members may also be under-cited, such as blacks, the disabled, junior faculty, conservatives, continental philosophers, non-western philosophers, and so on. Are there objections to referees employing this kind of strategy? Would you do it, and in support of whom?

Use innovative tools to teach clear and courageous thinking
Subscribe
Notify of
guest

29 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Lauren Ashwell
Lauren Ashwell
9 years ago

I wonder if a better way to influence gendered citation practices is to agree to referee those papers that don’t cite any women, and point out that this is the case in your report. In practice, I’ve found that authors are generally unaware of the fact that they’ve been gender-selective in citations, and are willing to look at this issue, and appreciate their awareness of this being raised. Even in those cases where the author is less receptive to matters of gender equality, as referees we can certainly influence them, and hope that they will be wider in their citations in the future – even if this is done begrudgingly in order to not have referees object to their citation list. Of course, we don’t want to reward those who don’t cite women with shorter review times (!!!), but if you’re a referee who is sensitive to this issue, I think it is a good practice to referee those papers and work to fix this, rather than to try to punish those who don’t cite women – as the next referee selected might not correct the author on this!

Matt Burstein
9 years ago

It seems to me that, if you want to effect change in this way, what you should do is accept those very papers and make a point of highlighting this problem (and the others that Justin notes) in the comments.

David Sobel
David Sobel
9 years ago

So far I like the initial proposal better than the proposed modification. It seems to me that, at least if enough people followed such a policy, authors are likely to become more self-aware of their own citation policies and to work harder to make sure that they are not overlooking or ignoring good relevant work by women. The effect of the policy, I think, would somewhat impact submissions generally and not just the one’s that are refereed by sympathetic folks.

Chris Frey
9 years ago

Ideally, one will referee papers in one’s area of expertise. And if one is overburdened, this seems like an obvious initial criterion to reject requests. But I’m not sure if it is a necessary condition for agreeing to referee a paper at all. For example, I was once asked to referee a paper on mereology. Though I’m interested in the topic it is not among the things I write about. The editor knew I was not an expert. I agreed (since I don’t receive that many referee requests), and it wasn’t that difficult. I ended up rejecting the paper because its principal argument was invalid. Topic specific expertise wasn’t needed. I suspect that sometimes, knowing too much about a topic might incline one towards being a bad referee since they will likely have a lot of commitments that the author disagrees with and it can be hard, for many, to recognize a good paper as good if one disagrees with first principles, etc.

Dale Miller
9 years ago

There are so many reasons that it can take a long time to get a verdict on the paper that I think it’s unlikely that authors in that position, who wouldn’t even be told that n people who were asked to be reviewers declined, would conclude that they must not have cited enough women.

anon
anon
9 years ago

Assuming that there is relevant literature by female authors readily known to the reviewer, would it be a fair practice to reject a paper or suggest a revise-and-resubmit based entirely on the grounds that the author did not cite any literature by female scholars ?

Anon2
Anon2
9 years ago

I can’t believe what I’m reading. Instead of judging a paper on its philosophical merit, philosophers want to push their agenda by punishing authors who don’t cite females. Yes, you are punishing them. By declining to referee an article because of “gender-selective citations” you are delaying the process. And if a referee recommends a revise/resubmit (RR) for a paper because of citation selection you are also delaying the process. How? I recently received a RR because, in part, of my citation list. So I had to read the material and figure out how to incorporate it. That took time. And for those of us on the market every day counts when it comes to publications. Here’s a novel idea: how about referees recommend revising citation lists when the material is *relevant* and *not* when there is some perceived gender bias?

Brian Weatherson
Brian Weatherson
9 years ago

This feels too roundabout to work. When authors get a paper back after a long delay, their usual reaction is “I can’t believe that referee took so long.” That’s sometimes combined with “and did such a sloppy minimal report”.

But what’s often the case (usually the case in my experience as an editor actually) is that the paper took a long time to get back because it took forever to find a referee. And the report was sloppy and minimal because the editors leaned on a friend who kinda knows the relevant field to write something quickly about the paper that had been sitting in the in-box for so long.

The point is that we would need to change a lot of views about what causes delays in getting reports back from journals before people started thinking that doing X, Y or Z was the cause of it taking forever to find a referee for their paper, and hence for their paper taking so long to come back from a journal.

Brian Weatherson
Brian Weatherson
9 years ago

Oh, and if as a potential referee, you’ve read the paper closely enough to have thought about what relevant work the author hasn’t cited, please just take a few minutes to write that down in a report rather than simply decline.

As an editor, I’d much rather get something that said “This is not a full report, but I noticed the paper has flaw X” rather than a simple decline. Some papers were borderline decisions whether to send to referees or desk reject, and even noticing one extra flaw might make a decision easy. And if the paper does deserve a fuller report, flaw X can be included in the editor’s note that goes along with the referee reports. Or, if your report says more about why X is a real flaw, it can be included as referee 2 or 3 or whatever.

It’s really hard to get referees for papers these days. Half a report, or a quarter of a report, is better than nothing, and maybe the editors can combine it with something else to make a full report.

anonpq
anonpq
9 years ago

Idea: decline to review such papers, but suggest as potential reviewers those relevant authors from underrepresented groups who have not been cited.

McCarthy
McCarthy
9 years ago

It’s going to be tricky figuring out which cited authors are conservative or not, but I guess I will give it a shot.

Jennifer Frey
Jennifer Frey
9 years ago

Is it really that hard to find referees? I’ve never turned down a request, but I only get about 4-6 requests a year. And yet I know some people claim they get that many requests in a week. Perhaps editors can expand their pool of potential referees? Or maybe if you are a big shot with tons of requests you can recommend young scholars who are perhaps being overlooked?

As for not citing women as grounds for rejection, I think I am less inclined to endorse that. Yes, it is extremely annoying that women’s contributions are undervalued, and this has all sorts of bad side effects. But another side effect (inevitable, I think) is a kind of reverse discrimination against women who are suddenly often cited. The claim will be: oh, she only gets cited so much because she’s a girl. This already happens to young women on the job market. I was once told to my face by a senior male philosopher at the “smoker” that the *only* reason I had so many interviews is because of my “chromosomes.” I wasn’t even by myself when he said this, it was said in front of others!

Perhaps a better (though no doubt naive and potentially hopeless) remedy is to continue to try to educate people about bias and to be aware of it in citation practices. We can start this awareness in graduate school. No one ever mentioned this when I was a graduate student, and we had workshops on publishing.

anon
anon
9 years ago

If you spend enough time on a paper to be able to say confidently that it is missing a reference to a particular paper (regardless of the gender of its author) then you are already in the position to write a review on it. If I was the editor, I would be rather upset to hear your reason for declining the review.

Rachel McKinnon
Rachel McKinnon
9 years ago

anonpq (sorry, the ‘reply’ button isn’t coming up for me): But then that will likely unduly burden people in those underrepresented groups. We’re already generally overburdened with service obligations because of our underrepresented status (e.g., women quotas for various committees, etc.). Given that there is a serious underrepresentation problem in citations and (more significantly) in whose views are discussed in the bodies of papers, nearly every paper submitted will have this problem. This means that on your proposal, we’d need to send nearly every paper to at least one referee from the underrepresented group(s). And since these groups are underrepresented, they’re not very big in comparison to the number of journal submissions requiring refereeing. So…perhaps you can now see the problem.

I already referee 10-20 articles a year. I worry that on your proposal, each person from the various underrepresented groups would need to referee many more than that. And that’s putting the burden on fixing the underrepresentation problem *on the underrepresented* people. That can’t be right.

anon
anon
9 years ago

The proposal is not a good idea. Surely what matters is whether there are relevant papers that have not been cited. Not every paper written in the area of the article that has to be reviewed is relevant, and therefore not every paper written by a woman in that area will be relevant. To be able to judge that the paper does not cite relevant articles one has to read the paper rather closely. And then one is in a position to write a report. And perhaps say on it that the paper does not cite relevant bibliography.

Jessica Wilson
Jessica Wilson
9 years ago

I’d like there to be a ‘Update the paper taking the neglected relevant literature into account and then submit’ option. Failure to properly engage with existing literature (written by men as well as women) is a basic problem of many of the papers I referee.

Rachel McKinnon
Rachel McKinnon
9 years ago

One worry I have is that I think our judgments of what is relevant is likely clouded by implicit biases and other ways that make our judgments unreliable. Rather than being conservative on what we think is relevant and deserves–at a bare minimum–citation, I think we should err on citing more. Perhaps this would allay some of the worries expressed on this topic. Part of the problem seems to come from a worry that citing something not clearly relevant (to the author, or objectively) is a bad thing. I’m not convinced about that. But I also think we can separate that question from whether it’s a good *policy* (principle, rule of thumb, heuristic) to, in general, be more liberal in what we’re willing to cite. Common to issues of rule vs act utilitarianism, good policies will invariably sometimes lead to something irrelevant being cited, but the overall outcome will be better. Moreover, something irrelevant being cited (when cited under the policy/rule of thumb) isn’t a big deal.

JDRox
JDRox
9 years ago

I am not against making a conscious effort to try to cite women and other underrepresented groups, although I am slightly uneasy about it for the reasons outlined by Jennifer Frey below. But I think the problem here is that there is some ambiguity in how the word ‘relevant’ is being used. If ‘relevant’ means ‘should be cited, regardless of sex’, then of course one should insist that such relevant papers be cited. But it isn’t clear that that is what is being suggested. Consider:

“In many such cases, it is easy for me to think of a paper by a woman philosopher squarely on topic quickly off the top of my head.”
“I am sorely tempted to adopt the policy of not refereeing any papers that could easily cite papers by women, given the topic, but don’t.”
“The idea, I take it, is that authors would come to know that it will be less difficult for editors to find referees for their papers if their papers cite relevant works by women”

The natural reading of these claims is that, if there are 5 relevant papers one could cite (with, say, one by a woman), and the most relevant paper is by a man, one should rather cite, or at least also cite, the paper by a woman. If this is the suggestion, I think it is ultimately misguided. Often, when some topic in a paper I am writing comes up, I want to cite some relevant literature. But often I have only read 2 or 3 papers in that literature. Let’s say that it is the ideal case, and I have read the 3 most relevant papers, all by men, although the 5th most relevant paper is by a woman. Still, I could easily cite that paper, even if I haven’t read it–at least if I know about it. (Which I may well not–this is not my AOS, just something coming up in a paper. This use of the word ‘easily’ is a bit weaselly.) But should I cite papers by women that I haven’t read just because I can? Or should I read the paper (instead of the the 4th most relevant paper) just because it is by a woman?

I fully endorse the idea that women and other underrepresented groups should get what they deserve, and people adopting various rules that enforce this norm.
I am sorta-kinda ok with the idea of women and other underrepresented groups sometimes, informally, getting a bit more than they deserve, for the purposes of achieving greater future equality.
I am not ok with the idea of explicit rules or norms that punish or otherwise disadvantage people who give women and other underrepresented groups what they deserve, but no more.

Jessey Wright
9 years ago

JD: The heuristic I adopt when I find myself in that situation (I want to cite a literature I recognize as relevant, but don’t know much about) is to, like you describe, read only a small selection of papers that I infer to be representative (which I identify by asking colleagues who do work in the area what is representative). However, when I choose those initial papers to read, I have a preference for reading papers written by women and members of underrepresented groups. In this way, I achieve both aims: (1) citing relevant literature and (2) citing women and underrepresented groups.

Also, several commenters have used the phrase “most relevant” which is a notion I find problematic. What is most relevant to a paper (beyond the core positions it is engaging with) is ultimately going to depend on what the reader is getting out of it. Relevance, I think, is as much determined by the reader’s perspective as it is by the author’s intent. Which is more reason to cite generously.

Overall, I think philosophers could use to cite more. As Rachel notes, the cost of citing something that isn’t really relevant is low. As an added benefit, larger citation networks benefit everyone. For those trying to get jobs, or tenure, if citation records matter then they’ll have larger citation records. For those trying to break into a new literature, more generous citation practices will make it easier to get a more complete view of the literature (and related literature) because a larger portion of it is cited. And, for those entrenched in a literature, more generous citation practices may led to you discovering something you didn’t realize was out there (and relevant) while reading papers you do recognize as relevant.

Jan Dowell
Jan Dowell
9 years ago

In the case described, the Potential Referee (PT) is getting 1-2 requests a week and is already committed to only referee papers on topics s/he has specialized expertise in. Even after that weeding, PT has more requests than can be accepted. The proposal is to decide among the remaining requests by declining to referee papers that don’t cite women and for which it is quite easy to identify a woman author.
I’d be curious to know what policy those who suggest PT should referee the papers that don’t cite women would recommend for deciding which ones to decline.

Dale Miller
9 years ago

One possibility would be the reverse policy of declining the ones that do appropriately cite women, so that s/he could take maximum advantage of the opportunity to correct the shortcoming in those that don’t. Another possibility might be to do the ones that s/he finds most interesting.

ABC
ABC
9 years ago

Jan: Does that one have to have a policy for those cases? Why can’t one decide on a case by case basis? In any case, I would choose to referee either the paper that I think I am better suited to express an opinion about, or that I initially find most appealing/interesting.

Crimlaw
Crimlaw
9 years ago

Here’s another factor that one might use to help make decisions about which referee requests to accept and decline: the price of the journal. Join the boycott of the most expensive journals. Go out of your way to help those journals that are free or as close to free as a journal can get.

Brian Weatherson
Brian Weatherson
9 years ago

Here’s a simple plan for deciding which to accept/reject.

* Decide how many papers you are willing to referee for a year, call in n.
* Divide the year up into n chunks. (Perhaps not equal; there could be a gap around exam grading, for instance.)
* In each of these chunks, accept the first paper you are asked to referee that you seem basically competent to referee having seen the abstract.
* Until the next chunk starts, decline the rest, preferably as soon as the email asking you to referee comes in.

I don’t see the need for a fancy decision procedure for deciding whether to accept/reject a request. Certainly I don’t see the need to read the whole paper; by that stage you may as well referee it. But the more important thing is to either accept or to decline quickly, and a simple counting rule like this seems like a rule that would help do that.

Rachel McKinnon
Rachel McKinnon
9 years ago

This is basically exactly what I do. Add one more: do not accept more than two pending reviews at one time. If I have two pending reviews, I automatically reject any further requests.

Chris Frey
9 years ago

I’m always surprised when I hear that people get multiple referee requests a week. I get maybe 5 or 6 a year. And it’s not as if I don’t publish. In fact, I publish in multiple areas. There are lots of people like me who are happy to referee more but simply aren’t asked. I don’t have any general recommendations for editors looking for referees. But there are many potential referees around if editors expand their search.

Nicholas Denyer
Nicholas Denyer
9 years ago

Sometimes an editor sends out to referees a submission that even an inexpert judgement could tell is not publishable. In consequence, I now always ask editors to confirm that what they ask me to referee is not like that. And if they do not give such confirmation I refuse to referee.

Jonathan D. Jacobs
Jonathan D. Jacobs
9 years ago

I don’t think that declining reviews will contribute to any feedback loop to authors about citation practices. If you decline, the editor just looks for the next referee. (And yes, it is hard to find referees. I often have 12-14 declines before I get two referees.) The only way feedback is going to get sent is if there’s a report. (Nor do I think that a delay in review on account of declined invites will send any message about reference lists absent explicit indication from the editor or referees that this is what caused the delay. The author will likely think instead that the journal is just slow.) So if the intent of declining reviews on this basis is to send feedback to authors about citation practices, I think the proposal is the wrong way to accomplish that goal. Writing a report that addresses citations practices, as many others have noted, Lauren Ashwell first, is probably the best way to accomplish that goal as a referee. (There may be *other* reasons to decline refereeing such papers, of course.)