Should Graduate Students Referee?


Should graduate students be called upon to serve as referees for journals? I was stunned a few years back to learn of the growing use of graduate students to serve as referees—stunned until I remembered the (arguably) over-publishing practice of our profession. But now the practice of enlisting grad-student referees—to my limited and aging eyes—appears to be gaining the feel of normalcy. This worries me but I wonder whether my worry is unfounded. Thoughts would be appreciated.

That’s Jc Beall, Board of Trustees Distinguished Professor of Philosophy at the University of Connecticut and Professor of Philosophy at the University of Tasmania, seeking opinions on having graduate students referee for academic journals. He continues:

I can think of some reasons to enlist grad-student referees:

1. So much publishing that there’s no alternative but to enlist as many recruits as possible.

2. Give the budding professional philosophers a sense of the refereeing life towards which they (as candidate professional philosophers) are heading.

3. Expose the grad students to cutting-edge ideas in the latest submitted drafts.

4. Give them a line (of what value?) on the CV.

Perhaps others can think of others. I note that (2) – (4) strike me as very, very weak reasons. (2) and (3) can be accomplished by an advisor—serving as a referee—via having the student meet with said advisor to witness (and engage in a less official fashion) in the process.

I can think of some reasons against enlisting grad-student referees:

5. They already have too little time for their own work. Why should they be given work that few want in the profession?

6. They are not yet fully in the profession, but are being asked to serve anyhow.

7. They might feel pressure to impress by their report—anonymous as it’s supposed to be. This desire to impress “the key holders” in the profession is not good for the peer-review process.

8. We may hope—for the sake of grad students—that grad-student referees are not high on journal-editor candidate lists. If that’s right, then (ceteris paribus) grad-student referees are being pulled as a last-ish resort to do what the profession considers to be very important work. (And maybe they’ll do a good job, possibly better than those already in the profession. But why should they be doing it at all?)

Perhaps others can think of others. I note that, in my view, (5) and (6) are strong reasons to steer very clear of using grad-student referees; (7) is strong if applicable; and (8) ultimately turns on empirical data concerning the process.

My question, again, is just this: should we support any grad-student refereeing in our profession? And if so, in what sort of special case(s)? (Specialization and the just-finished-dissertation-on-the-topic case will be relevant, I think. But even here, why burden the student with the official report rather than have them give the advisor their thoughts?)

Readers?

UPDATE (6/2/17): The Chronicle of Higher Education looks into the question.

guest
31 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Michael Oxenrider
Michael Oxenrider
4 years ago

What makes it “over” published? I’m being skeptical _and_ genuinely asking.Report

Eric
Eric
4 years ago

Another reason to enlist grad students: They submit papers to journals. They therefore contribute to the enormous number of papers that need refereeing. In most cases, the students asked to referee are students who already have publications or who are far enough along in their program to have a paper or two under review. Seems reasonable to expect them to take some of the refereeing load, even if most of it placed on fully-employed philosophers.Report

B
B
Reply to  Eric
4 years ago

I agree that if you contribute to the work load, you should contribute to the work. I also think that those who are un-employed or underemployed should adjust their share of the work load accordingly. Those who have not really been accepted into the profession should not feel a special onus to alleviate the problems of the profession.Report

Ch
Ch
4 years ago

As another ‘pro’, how about the fact that a graduate student in the middle of a dissertation project usually has a far better grasp of the literature as a whole than senior faculty?Report

Cautious Until Tenured
Cautious Until Tenured
Reply to  Ch
4 years ago

Indeed. It seems that there is a presupposition here that peer review is the province only of those with the professor title. But the real requirement is that the review be by a PEER – where peerhood ought not be measured by academic rank, but by relative expertise in the area in question. Many graduate students are the peers of many professors in many areas. Thus, provided that the graduate student has attained the sufficient level of expertise in the relevant area, there is every reason for them to referee for a journal, and no reason for them not to.

Of course, editors are under an obligation to make sure that the referees they are relying on are experts in the appropriate way. This is true no matter what the rank of the referee.Report

Jc Beall
Reply to  Cautious Until Tenured
4 years ago

Hi, Ch and Cautious Until Tenured — there’s no presupposition or even presumption that titles/ranks matter. I agree that ‘peer’ in ‘peer review’ is key, and that specialization makes the matter difficult. (See my final parenthetical comment on this, which was intended to flag the point.) Perhaps part of the question is whether ‘peer’ in ‘peer review’ goes beyond subject matter to standing in the profession — and, in particular, to opportunity (and opportunity costs) for would-be referees. Some have more opportunity than others. The question is in part time, opportunity, and what the profession should demand. I think that Eric (5.31.17 @ 7.36am) makes an interesting case — if one’s contributing to the pile of to-be-reviewed then thereby one has a prima facie duty to referee. (I’m not sure I agree, but it’s an argument.)

Thanks for your help in trying to think through this. JcReport

James Beebe
James Beebe
Reply to  Ch
4 years ago

I’m in favor of the idea that grad students can be the epistemic peers of professors. But the idea that they “usually ha[ve] a far better grasp of the literature as a whole than senior faculty”–i.e., that they are usually professors’ epistemic superiors (rather than peers) with respect to the topic of their dissertations–is a bit strong.Report

Kai von Fintel
4 years ago

At S&P, we use graduate students as referees sparingly but unapologetically. If I need a second reviewer on, say, embedded imperatives in Slovenian, and it turns out that one of the world’s foremost experts on that is an advanced graduate student, … it would be a disservice to the field not to call on that expertise. Report

Dale Miller
Reply to  Kai von Fintel
4 years ago

This is my practice at Utilitas as well. I could probably count on one hand the number of times that I’ve used grad students as reviewers in the last year. And I wouldn’t feel comfortable asking two grad students to review the same paper. But in some cases I think that the reasons in favor outweigh the reasons against. Another argument in favor is that having the experience of serving as a reviewer may put them in a better position to produce work that their own reviewers will smile upon.Report

Jc Beall
Reply to  Kai von Fintel
4 years ago

Thanks, Kai and Dale. This is what I think is inevitable, and I hope that other editors have your position: namely, ask when truly necessary given topic. (I guess I fear that, from not tiny anecdotal evidence, grad students are being pulled in not when the subject demands it — as in your cases — but when others won’t step up.)

Dale’s argument for is interesting, though I’m not sure the student needs to have the burden of actually serving so much as perhaps actively engaging with her advisor (or whomever) who takes on the official burden.

Eric (see above 5.31.17 @ 7.36am) makes an interesting argument.
JcReport

Alex
Alex
4 years ago

(5) and (6) strike me as especially *weak* reasons against grad-student refereeing. After all, if any of the considerations mentioned in (5) and (6) apply in a particular case — which is up for grad students themselves to judge, not something that one could know to be true of all grad students a priori — then that person has the right to just say “no”, and will be able to do so with absolutely no consequence whatsoever to their careers. Report

Dale Miller
Reply to  Alex
4 years ago

(8) strikes me as not a reason at all, at least not a reason against. If editors are turning to grad students as a last resort now, then what would they do if this were no longer an option? Write to authors and say that it won’t be possible for the journal to consider their papers because suitable reviewers could not be found?Report

cw
cw
4 years ago

@Editors: has anyone noticed a problem with the quality of reports received from grad student referees?Report

Kenny Easwaran
Reply to  cw
4 years ago

I don’t find a problem with the quality of grad student referee reports. But I do find a general trend that people who have written more referee reports tend to be more charitable, and people who have only refereed a couple papers tend to be more negative (though this is only a broad average).Report

Jc Beall
Reply to  Kenny Easwaran
4 years ago

Hi, Kenny. Thanks for this contribution to the discussion. Your findings sit well with my own experience over the last few years — for what that’s worth. Cheers. JcReport

Kate Norlock
Reply to  cw
4 years ago

At FPQ, I found that on the whole, grad students’ reports are usually as good or wanting, as mixed a bag, as tenured profs’ reports — but unfortunately, they skew recent! I.e., on the whole, I find that grad student referees tend to identify failures of literature-appreciation from the last ten years or so, but tenured profs regularly point out literature from thirty, forty, or fifty years ago that is not necessarily free on Google Scholar but that really makes almost all the same points as the submission, made a contribution in its time, and ought to be taken into account. There’s a palpable difference in what I can only call the arc of scholarship over time as represented in the different reports.

I was a bit shocked when I started editing a journal and people regularly declined to referee but then identified their graduate students as alternatives. I started out with the assumption that we should spare grads the service obligations. I’ve come around to the obvious goods of asking a grad student when they have more expertise on a particular topic than any tenured prof. But I continue to try to hold the line on exploiting grad students sparingly. That they contribute to the amounts of submissions doesn’t seem to me a compelling reason to expect reciprocal refereeing. After all, my students also contribute greatly to my marking load, and there are good reasons that they shouldn’t do the marking.Report

Jc Beall
Reply to  Kate Norlock
4 years ago

Thank you, Kate. This is important information (first paragraph); and your practice (2nd para) is in keeping with what I’m calling the Kai+Dale practice (see thread above). Thank you. I find the whole issue a difficult one to navigate, and the discussion continues to be helpful to me — and, I hope, others. JcReport

Anthony Fernandez
4 years ago

I support the practice for some of the reasons listed above, but for another reason as well: I began refereeing around the time my first article was accepted for publication, and refereed five or six articles before completing my PhD. I learned a lot from writing these reports—not so much about the state of the field, but about how to write for referees. Prior to refereeing, I read plenty of papers for colleagues, offering them constructive feedback where I was able. But reading a paper as a referee turned out to be something different—and something I hadn’t expected. When I referee, I’m not reading a paper as a supportive colleague or friend; I’m reading it to decide whether it should be accepted into this journal. (This isn’t to say that I don’t offer constructive feedback. I always write detailed and [I hope] constructive reports, even when I recommend outright rejection.) By refereeing, I quickly became aware of what not to do when writing for publication, because all kinds of new things stood out to me. And, in many of the online submission systems, I had access to the reports of the other referees, through which I learned not only what makes for a good (or bad) referee report, but also what impresses (or troubles) other referees—and, perhaps, should impress or trouble me.

In short, the act of refereeing didn’t just teach me how to referee; it taught me how to write for publication. Anyone writing for publication deserves that opportunity, whether or not they have a PhD.Report

Lelia
Lelia
4 years ago

I’m a fifth-year graduate student in linguistics and have reviewed for 2 journals as well as several conferences. I enjoyed the opportunity to review because it has helped me to develop an “imaginary reviewer” in the back of my mind as I write my own papers. I also believe that I was able to offer substantive and constructive comments, which often largely agreed with comments submitted by other reviewers of the same paper (who were probably not graduate students). I found these experiences to be educational rather than burdensome. The OP seems to suggest that vulnerable graduate students should be protected from the burdens of reviewing – but that is not the way I see it myself.Report

Jc Beall
Reply to  Lelia
4 years ago

Thank you, Lelia. It’s a complicated issue, and the perspective you offer is important and helpful. I guess the issue isn’t whether there are any benefits but rather whether the benefits can be achieved without the burden. (And it’s true that the first opportunities might not seem like a burden, but as editors will attest there are many candidate referees who’ve come to see the job as a burden.) Kai and Dale’s practice strikes me as a good one, overall, but it sounds like you might disagree with their practice of “sparingly” enlisting graduate students. Thanks again for helping me — and, I hope, all of us — to think through this. JcReport

Eric Schliesser
Eric Schliesser
4 years ago

I would look to put some flesh on (6). PhDs are being shortened temporally (that is increasinlgy less time to do research) and, as it happens, literature reviews are disappearing from dissertations. This means that, all things being equal, the current generation of PhDs are more like to be biased toward recent literature and somewhat more parochial in their understanding of it. (Not all things are equal, it is MUCH easier to do research these days than it was 20 years ago!) What makes an experienced, referee # 2 so useful is that s/he is more like to spot that an apparently fresh idea Y is really a recycled version of old idea Y* that disappeared from consideration and call attention to relevant literature.
As I said before not all things are equal, and I am not suggesting that this considerations trumps all the others, and I know of cases where PhD students are de facto the best person to ask.
Report

Jc Beall
Reply to  Eric Schliesser
4 years ago

Thank you, Eric (Schliesser). This is a helpful observation. At the very least, the consideration might add force to the Kai+Dale practice (see above), namely, that (post-) grad students be asked sparingly and only when their expertise is clearly needed. (On the other hand, some of the other comments raise interesting points in favor of normalizing regular grad-student refereeing. I remain unsure, but the discussion is certainly helpful so far.) Jc
Report

Manny
Manny
4 years ago

I am a third year graduate student. I was asked to referee a paper. The paper was very closely related to a paper I had recently published (as it happens, in a better journal than the referee request came from).

Given how closely related it was to my own work I felt well qualified to referee the paper — I’d spent the best part of a year thinking about just those issues. I also think it would be a shame if the journal had opted for someone less familiar with the material, simply because they had a PhD and I did not. So I think (8) is perhaps unfair.

And, given I have been reviewed myself, it seems only fair that I contribute back when I am qualified to do so. So I do not agree with (5) and (6). Report

Jc Beall
Reply to  Manny
4 years ago

Thank you, Manny. This perspective is helpful. I note just one thing, namely, that I never intended the question to revolve around having a PhD (or tenure). (James’ PhD Octopus is still alive and growing far too big!!) The question is in part an opportunity-cost question — in part. And even if the Kai+Dale practice becomes the norm (vs grad students seen as being in the exact same professional pool of candidate referees), there will likely be cases such as your own that call out for special consideration. I don’t think that the profession can come up with absolutes here (though I do believe in absolutes in general!); my hope is just to get some clarification and reflection on the practice, and already your comments and those of others are helping a lot. JcReport

Trying to Avoid Revealing Info
Trying to Avoid Revealing Info
4 years ago

I am an editor at two journals, and I have used graduate student referees about two or three times — with the main aim of providing the students with some training in refereeing (something I sorely lacked myself). These were my own ABD students, and I gave them feedback on their reports after submission. What I didn’t tell them is that in at least one case, I had already decided to reject the paper. Although using a student referee in this way is a little, er, nonstandard, I hoped that it provided suitable hands-on practice, while offering authors more response than they would otherwise have received. And because my students wrote sensible, charitable and helpful reports that also saw the flaws in the paper, I was able to use their refereeing as examples of their fine judgment when writing them letters of recommendation later. But really, my main goal was to give them some practice, so they wouldn’t feel as lost as I did when I started refereeing — and of course, to set them up so I can call on them to referee in the future!Report

Stuart
Stuart
4 years ago

I think journal refereeing is an important part of our job. Graduate students are (often) training to Become professional philosophers. Hence, they should be mentored into refereeing. I have occasionally been asked to review an article that is in one of my graduate students’ areas of expertise. On such occasions I have sometimes asked the editor if the official report could be done by my student, with help from me. Not only do I think this is acceptable, I think we fail our students if we don’t mentor them appropriately into this aspect of the profession. Report

JR
JR
4 years ago

I do not have a PhD and I am not a PhD student. Yet, I have been asked to referee a paper three times (including a top 20 general philosophy journal). Only once I have declined because I believed I knew the author. I think I did a good job and I did the reviews quickly since I did not have teaching duties etc, so I was able to prioritise refereeing. I hold a Master’s degree in Philosophy and I have published an article in a top journal of my AOS.

After completing one review, I got the feeling the editor did not even know who I was when asked to referee a paper. That is because at the time I had a paper under review in a triple-blind journal and during the review process, I was asked to referee a paper on a similar topic for the same journal. So either the editor did not know my identity when asked to referee or violated the triple-blind policy since I had paper under review myself.Report

Tim
Tim
4 years ago

I’m not a journal editor, so my anecdotal evidence may be totally misleading. But extrapolating from my experience most of you need a reality check. Before arguing whether they should do referee work, we should find out whether they already do it. From the comments here one could think that they rarely do it, but asking around in my department I found that almost all my colleagues started reviewing papers before finishing their Ph.D.
Also: Apparently editors think that 90% of the papers submitted to their journal are not worth publishing, but that for each paper – including those in the bottom half – they need tenured professors to tell them whether it sucks. Come on, you don’t need world leading experts to find obvious non-sequiturs or whatever you think 90% of the papers submitted to your journal suffer from. At least, if you have two reviews, one of them a graduate student, telling you that the paper sucks, you shouldn’t worry whether the graduate student’s assessment suffers from recency bias or general lack of expertise. For that is what happened most the times I reviewed a paper as a graduate student (assuming that the other reviewer wasn’t a graduate student, too). Report

Mike Titelbaum
Mike Titelbaum
4 years ago

On multiple occasions I have taken a referee request sent to me and asked one of the graduate students in our department if he/she would be interested in doing the report. I only do this with very capable, dissertating graduate students who I think know the relevant literature inside and out. I also always check with the journal editor first. Once the grad student completes the report, I go over it with him/her and almost always suggest changes. I’ve never viewed this as burdening the student. Instead, I figure that as professional philosophers they will eventually need to know how to write a thoughtful, substantive, fair referee report, so it’s worth having them learn to do so in a supervised environment. (Hence having them create the report themselves rather than just observe my creation of one.) Also as many commentators above have noted, refereeing gives you an invaluable perspective on how your own submissions will be evaluated. Of course this practice of mine is distinct from the idea of editors sending out referee requests directly to graduate students, but I thought the info might be helpful.Report

Jc Beall
Reply to  Mike Titelbaum
4 years ago

Hi, Mike. Thanks very much for this comment. Your practice strikes me as a useful approach — the sort of dissertation advisor’s Kai+Dale (editorial) practice. Cheers. Report