Journal Articles: Quantity & Quality
“Considering my own area of philosophy of language and mind, I don’t think there is all that much difference between most of what gets published in the ‘top’ journals, and most of what gets published in the ‘tier 2 ‘journals. My sense is that there is rather too much good work to keep track of, not that the difference between the top tier and the tier 2 journals is so great.”
(photo of untitled sculpture by Tara Donovan)
Following up on last week’s post on Robert Goodin‘s remarks about journals, a philosopher writes in with some critical remarks.
Goodin had said:
[The] blogsphere is full of bleating that journals should increase the number of articles being accepted, on the grounds that ‘my article was no worse than the worst article appearing there that year’. In most cases that is simply untrue. (I do freely admit, however, that in some years JPP published an article or two that we would not have done had we not been contractually obliged to publish a certain number of articles per year.) But suppose the claim were true, and just do the maths. Publishing more articles that are equally bad as the worst articles published would automatically reduce the average quality of articles in the journal—that is an unavoidable fact of simple arithmetic. Of course, each author of an almost-as-good article has an interest in their own article being accepted—but only theirs. If every almost-as-good article were published, the value of publishing in the venue would nosedive, to the chagrin of all authors publishing there.
Preston Stovall (University of Hradec Králové) sent in the following reply:
I was glad to sign the petition against refereeing for The Journal of Political Philosophy when it was sent around, and I stand by that decision. Business interests in revenue generated from published essays should not dictate editorial decisions at a scholarly journal. But I’m troubled by Goodin’s remarks about the “bleating” from people who call for an increase in the number of publications in top journals. I’m not sure the line of thinking here is right, but I can’t shake it, and if there’s something wrong with it I figure someone can point it out.
Considering my own area of philosophy of language and mind, I don’t think there is all that much difference between most of what gets published in the “top” journals, and most of what gets published in the “tier 2” journals. My sense is that there is rather too much good work to keep track of, not that the difference between the top tier and the tier 2 journals is so great. And more generally, given how many PhD’s are being produced each year, and owing to the strengths of the teachers and programs producing them, I find it hard to believe that the absolute numbers of publishable essays, at any venue, aren’t increasing.
That is to say, supposing whatever you’d like for the meaning of “top journal”, and given the increase in number of journal submissions everywhere, I don’t see the basis for thinking that the relative number of submitted to publishable essays at the top journals has dwindled so far, and so fast, that the top journals would decrease the value they add to the scholarly community by publishing, say, 50% more work than they currently do in a year. At least, not for what’s being produced right now in the philosophies of language and mind. Maybe political philosophy is different, but I don’t know why it would be. And I worry that in practice this becomes a way the profession ossifies around cadres of insiders gatekeeping their fields and subfields.
Either way, I hope that anyone who does hold this view is either at a department that does not have a PhD program, is actively trying to reduce the number of students admitted to that program, or is actively working to see that the program prepares graduates for work outside the academy. A job market producing an increased number of candidates whose work is regularly judged to be of poorer relative quality than previous generations, and at a time when judgments of value concerning things like publication venue are becoming so important for landing a job, is doing a disservice to the people in the field who are the least well off.
Discussion welcome.
Without wading into the substance here, I just wanted to isolate what I think is a key formal model-ish assumption in the new argument here, to maybe draw one sharp distinction between the two sides of argument:
That “quality” papers are treated as scaling with the community’s total throughput/papers coming out are “graded on a curve”. (So, as the rate of total papers increases, so too does the rate of quality papers). What does this amount to? Maybe quality is an intrinsic property of papers, maybe it’s a property of the community’s statistical draw from a collection of all possible theses, or something else. But then under the modest view that quality papers should see the light of day, we get the rest of the argument following from that basic modeling setup.
But maybe “quality”, considered at the level of individual papers, is mostly a red herring. I understood part of the previous argument to be that there is value in a journal decoupling the question of a paper’s quality from the question of the community’s benefit of having that paper, together with others, put in front of them. Lottery funding models with a threshold are maybe an example of this alternative mechanism, and maybe that association (linking the present discussion to a context where economics of funding really is what matters) explains the distraction that has sometimes occured, where people have pointed to economic considerations about “available journal space”, “audience’s time to keep up to date on the field”, etc in setting up the positive argument for severe restrictions semi-independent of paper quality.
Here’s something Goodin said that leads me to think considering quality at the level of individual papers is not a red herring:
Maybe JPP in particular, or political philosophy on the whole, is unlike my experience with material being published right now in the philosophies of mind and language. And maybe the material published in political philosophy is such that it’s easy to see whether a paper has been published in a “top” or “tier 2” journal. But I don’t see why it would be, and I can’t shake the suspicion that it’s on balance bad for the profession to frame things in the way Goodin has. Which, again, is not to call into question the justification for refusing to referee for JPP given the way Wiley behaved.
Right, thanks for that reply. I think I am now convinced that paper quality is not a red herring in the dialectic here, because of assertions of judgement being made in the original as partial support for Goodin arriving at the view offered. But I’m more ambivalent/remain ambivalent about whether it is a red herring in what the argument (ought to) boils down to, which I think is exactly on the point you say: “cant shake the suspicion that it’s on balance bad…” I think that’s where the action of the argument really is, on whether what is good is journal practices reflecting facts of quality versus something else, something like stewardship maybe? (Not sure exactly)
This is just my opinion, but I do some work in political philosophy and I don’t really think it’s all that different. For what it’s worth, I think that “tier 2″* journals in political philosophy have published some work that’s just as good than anything that’s been published in JPP, PAP, or Ethics and quite a bit better than the average paper in any of them. For instance, I think Robillard and Strawser’s “The Moral Exploitation of Soldiers” is an incredibly important and good paper and it appeared in Public Affairs Quarterly. Or to take a direct comparison Mika Lavaque-Manty’s “Kant’s Children,” which came out in “Social Theory and Practice” is one of the best pieces I’ve read on Kant’s views on the moral status of children and the ways it ties in to its larger political philosophy. I won’t name names but I remember noting at the time I read it how much better it is than a paper on the same subject that appeared in a tippy top journal around the same time. Look I back Goodin’s decision– we can’t have publishers meddling in peer review– and I’ll note that he did fine job at JPP, but this idea that there’s a finite pool of papers of sufficient quality and he couldn’t publish more without diluting JPP’s quality just doesn’t wash to me.
*By tier 2 I mean journals that are in the bottom of the top 25 in Leiter’s poll. Personally I don’t think this “prestige” measuring that Leiter and so many other philosophers are obsessed with is helpful in any way and dislike “tier 2”. But I’ll go ahead and use the term for the sake of argument.
But is the existential claim the interesting one? I have no doubt that there is great work outside of The Top Journal and Nearly The Top Journal. But: if I only have so much time, and I want to flip through things, where am I more likely to find something that is valuable to me to read? That the might be something great out there doesn’t mean I should do some reading explorations out there. There are serious opportunity costs!
Assume, for a moment, that referees are not wholly incompetent. There are some false positives, and there are some false negatives, but on the whole, referees (it seems to me) do a better-than-luck job. If we then also assume that authors would rather be published in The Top Journal rather than “East Coaster’s Blog,” we have a ready explanation for why we might see higher quality _on average_ in The Top Journal than in “East Coaster’s Blog.”
In all honesty, I stopped thinking of articles in terms of “Journals” a long time ago. When I want to read up on something for a paper I’ll first go to Philpapers and just search around for the topic, read abstracts until I hit a few things that sound like they’re on target, and then specifically read a few of those. Sometimes those lead me, via footnote trails, to other good papers. Failing that, I’ll look through Google Scholar.
I’m genuinely surprised to hear that people still “read” in terms of journals. I can understand, for tenure or job reasons, to want to publish in specific journals but I don’t really see the point (especially if we’re after efficiency) of reading through journals.
I have to admit that I’m pretty much in the same boat as Caligula’s Goat. But I’d add that average quality can mean a lot of things. It seems that Goodin means that he believes pretty much every paper in top journals is very good and practically none in second tier journals are of the same quality. You seem close to that (correct me if I’m wrong) but grant a few more excellent papers get overlooked by the “best” journals and end up in the “tier 2” journals. That just seems false to me. The floor is so high these days that you just don’t really see flat out bad articles in any reputable political philosophy journal. But contra Goodin I tend to think that there really isn’t that much of a gap between the average article in “tier one” and “tier two” journals. Most are pretty good but also unexciting to be honest. I do think that more really excellent articles that are exciting appear in the top journals but in my rough estimate it’s really more like something like a little bit less than 1/5 for the tippy top journals and 1/10 for the others. It’s not enough of a gap that I am at all confident that I can ignore stuff published in tier 2 journals or that I’m sure I’ll find anything that’s worth the opportunity cost (as you say) of reading in any given issue of a top journal. I’d also when does run across one of those 1/10 it often is even more interesting and exciting than the excellent papers from the “top” journals. I guess it just seems that the “tier 2” journals are willing to take more risks and bit more likely to publish more stuff that’s weird in a really good way. That’s one of the reasons I pretty much do as Caligula’s Goat does these days.
I find value in the journal-publishing process in that, (a) it gives me a hurdle/bar that I have to reach – sharpens my self-criticism during the writing process, (b) the reviewer feedback is useful… sometimes.
However, I think there’s less value in journals as “gate-keepers”/ quality control. (a) quality control itself isn’t so important. Philosophy is bound to be 99% dross, it’s worth having more dross for the few extra diamonds – throw everything at the wall and see what sticks. Too many false negatives (i.e. everything Nietzsche wrote would be desk-rejected) and false positives (i.e. the kind of paper that masters the current literature/lingo, but fails to make contact with the actual subject matter. (b) in the internet age, it seems like there are ways of doing post-publication quality control, i.e. rather than 1 or 2 people controlling what sees the light of day, philpeople could have a “Billboard 100” type system – users anonymously upvote and downvote papers (just one of many technologically feasible suggestions). I feel that the format of teaching and research are on the edge of a revolution, that the “legacy media” and its associated prestige are holding things back, but about to wilt.
Stovall’s remark is about the value the journal provides by publishing papers. Goodin’s remark is about the value to authors of having a paper published in the journal. The two are talking about different things.
More to the point, the value to authors of having a paper published in a particular journal can decrease because of a decrease in the average quality of the papers in it, even when there’s no decrease in the value a journal provides to the scholarly community. For example, plausibly, the value a journal provides depends on the number and quality of the best paper it publishes. This number and quality need not decrease when the average quality of the papers the journal publishes decreases. However, the value to authors of having a paper published in that journal can still decrease significantly if the average quality decreases significantly because, as a result of the latter decrease, having a paper published in that journal would no longer be percieved as a significant achievement. If having one’s paper published in the journal is no longer a significant achievement, the value of having it published in that journal diminishes.
Here’s what Goodin says (p.11-12 of the interview).
I am denying, until such time as I have reason to think otherwise, that the publishing landscape is well-represented by framing it in these terms. There is more “top” quality work to be published than there are places to publish it, and journals that strive for selectivity in this market, of a kind that justifies itself on the basis of the supposed disparity between what gets selected and what doesn’t, are doing a disservice to the profession — to authors, to journals and their readers, and to the public that funds us.
It also strikes me as a bizarre way to look at the role of journals in Anglophone academic philosophy today. I try to keep up on what is published in my field, in basically any journal I’m familiar with. And I can’t tell you how many times I’ve either read something in a “top” journal that was a stinker, or read a very good essay in a “lower-tier” venue. It’s happened often enough that I know not to give a lot of credence to venue. Again, maybe political philosophy is different. But the view that grounds this approach toward journal editing strikes me as jarringly out of step with the field today. I might be wrong, but I don’t see where.
My guess is that you two disagree about the current average quality of papers published by top journals. Perhaps you think the average is fairly high, for you say “[there is more ‘top’ quality work to be published than there are places to publish it.” But perhaps Goodin thinks it’s fairly low already, so that diminishing it even more would be a serious disservice.
For what it’s worth, I don’t agree with your assessment of how much top quality work there is. Unless top quality is simply defined by what gets published in top journals. Much of what gets published by those journals isn’t of a high quality.
My argument doesn’t turn on any particular conception of what “top” means, and I’ve signaled that by using scarequotes throughout. All that matters is the similarity between what is published in the “top” and the “tier 2” journals, together with a putative justification for selectivity made on the basis of a supposed difference in quality between what gets selected and what doesn’t at the “top” journals. As I said, I think that stance is doing a disservice to the profession in a situation like we have today. Still, I’m open to changing my mind if someone gave me good reason to do so, and maybe political philosophy isn’t like the philosophies of mind and language. (And again, this doesn’t detract from the opprobrium owed to Wiley over their treatment of Goodin and JPP.)
I think I understand. You judge the quality of papers at top journals to be extremely similar to that of the papers at tier two journals. This is why you believe there’s more top quality work than there are places to publish it. And this is why you take issue with Goodin’s remarks. If that’s correct, I suspect the source of your disagreement with Goodin is your judgment of similarity of quality.
If I’m right about the source, there’s likely no way to get you to change your mind and recognize the error of your view. Doing so would require you to become able to identify the differences in quality that there tends to be between the papers published in top journals and those published in tier two journals. But if you’re unable at this point, you’re unlikely to become able.
Yes Sam, I’ve signaled throughout that my experience with the literature in the philosophies of language and mind right now is that there is not much difference between what tends to get published at “top” and at “tier 2” journals. I also gave an argument in the OP to the effect that we don’t have good reason to think the general quality of submissions has plummeted so far and so fast, given how many PhD’s are coming out of good programs with good teachers. I’ve also made clear that I’m willing to change my mind on the basis of some reason to think I’m wrong about something. So far, no reason has been given.
A possibility that you don’t seem to acknowledge or take seriously is that much of the work published in top journals isn’t of high enough quality and doesn’t merit publication in a top journal. Once you take this into account, confidence in one of your unstated assumptions should be decreased: that similarity of quality between papers published in top and tier two journals indicates that there’s more top quality work than places to publish it (here by “top quality” I mean work that merits publication in top journals).
Recognizing that the possibility mentioned is an actuality should give you reason enough to recognize your error. But if you’re unable to identify the differences in quality at this point, you’re unlikely to become able.
We’ve been given no reason to take your “possibility” seriously, no reason to think the argument I gave about the quality of work being produced today is flawed, and so no reason to revise the conclusions I’ve drawn. And if one were to take your “possibility” seriously, then as I said in the final paragraph of the OP I hope anyone with such a view is doing one of the three things I called for there.
This all seems evident enough to me that I’m content to leave it at that. Feel free to have the last word.
Once you make your argument clearly, I promise to point out its flaw and thereby give you a reason of the sort you’d like. Put your argument in standard form.
If that’s right, then Goodin is even more wrong than Preston thinks he is.
I think I know why you say this. But I’m not sure. Care to elaborate?
Because your suggestion is that Goodin is wrong about the quality of most of the work in his journal, which is actually much lower and doesn’t merit publication in JPP.
Preston’s suggestion is much more modest: it’s that there’s lots more excellent work in political philosophy than Goodin’s favourite ten (or whatever) submissions.
Thank you Preston for a very interesting post! It suggests that there are (at least) two main ways to rank articles:
(1) Articles are ranked against a fixed permanent ordinal scale of quality. If this system were used, journals would publish however many articles surpassed some threshold. That number would be variable.
(2) Articles are ranked using a relative-strength ELO system. If this system were used, journals would always publish a fixed number of articles, no matter how excellent they might be on an ordinal scale.
Since the number of submitted articles is increasing, and journals are not increasing the numbers they publish, I suspect that the ELO system is in use in philosophy. Of course, the ELO system doesn’t measure ordinal quality; it does measure a sort of Darwinian competitive fitness, which might even punish ordinal quality.
It might be interesting to use LLMs to provide an ordinal measure, which would at least have some degree of standardization. Here’s a suggestion for using LLMs for article review: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10191680/
I read the regular online threads calling for more publications, and I read the stack of essays I’m asked to referee. The threads are filled with people saying that there is lots of top-quality work that the journals are keeping out. My stack suggests otherwise.
As I’m reading, my refereeing guide is not “Do I agree with this essay?”, nor is it “Do I think this essay could be better?”. Rather, it is “Do I think that my colleagues, with limited time, would benefit from reading this essay?”.My colleagues want to be provoked to think a new way, they want to be exposed to a new line of arguments or a new scholar, and they want existing discussions carefully reframed. Now, my judgments might be flawed. But: I reject more than 90% of what I receive. So much of what I receive is recreating a very-well known wheel, wildly mistaken about the state of the existing arguments, or offering a quick argument without considering obvious concerns. My colleagues don’t have time to read that sort of thing. And they don’t have time to sort through stacks of hay, poorly written hay, in order to find a golden needle.
To be clear, I’m comparing what is published in “top” and “tier 2” journals, not what is submitted to them. And it can both be true that there are many more unpublishable essays being submitted to top journals today, and that the essays that tend to get published in top and tier 2 journals tend not to differ much in quality. It’s certainly true that I recommend rejecting more essays than I recommend accepting on the first submission. But that doesn’t entail that there aren’t many more acceptable essays being submitted to journals today than, say, 30 years ago. And I don’t see how that situation isn’t to be expected, given how many well-trained PhDs are being churned out today.
This is a good distinction. (Though it would be a stark understatement to say that I recommend rejecting more than I recommend accepting.) But, perhaps contrary to your experience, and perhaps our disagreement is about what counts as “top” or “tier 2,” I do see a noticeable difference, if not a uniform difference, in quality.
Also, from the armchair, I agree that there are lots of well-trained PhDs being churned out today. On the other hand, the pressure to publish, and to publish early, is sky high. I couldn’t say whether the quality of submissions or the relative number of great pieces has gone up. There is, to be sure, _lots_ of great philosophy now.
That’s all fair. I don’t think my position requires supposing publications in “top” and “tier 2” journals are across the board indistinguishable. My own experience is that I’ve seen enough material fit for the other category as to make journal venue a rather unreliable guide to publication quality, but it’s compatible with my position that others find venue more reliable. And the point about the pressure to publish is well taken. Still, my sense is that, as you say, there’s lots of great philosophy produced today, and enough of it gets published outside the “top” journals as to make a position like Goodin’s seem rather misguided.
I think this is a really interesting question, but I think that exercising one’s own judgment about work in one’s own specialty is likely to be misleading. The reason “second tier” specialty journals exist is because specialists are so interested in a topic that they can find huge interest in work on that topic that just objectively isn’t as interesting as the most interesting stuff in the field.
How would interestingness objectively?
Also, although one’s judgement of the work in one’s own specialty may be limited and biased in some senses, is this perspective not a highly important one? What do the pros and cons of relying of one’s judgement of one’s own specialty imply in the context of how this post also involves judgement of another field?
I typed the original comment quickly on my phone. I meant “How would interestingness be judged objectively?” and at the end of the last question I meant specialty rather than field.
I’ll push back against this, as I didn’t take myself to be talking about material in the literature on mind and language that was of interest principally to specialists.
My sense is that there is insightful work being done right now — particularly at the interstices of philosophy, cognitive science, and linguistics — that’s worth being taken seriously outside the specialized areas in the philosophies of mind and language producing this work: e.g., research on shared intentionality in philosophy, psychology, and primatology, and syntactic or proof-theoretic resources increasingly put to use in programs in the philosophy of language, metaethics, and linguistics. Much of this is being produced outside of the “top 10” or “top 20” journals that tend to crop up in polling, even when written for a non-specialist audience. There’s work being done in these places that’s of wide interest in addressing questions in social and political philosophy, the philosophy of perception, metaphysics, philosophical logic, and the history of analytic philosopy (to rattle off a quick list). Collectively, this work forces us to re-evaluate many of the founding myths and merely ready-at-hand tools Anglophone philosophers tend to reach for in their work.
I don’t read as widely as I’d like, but my sense is that things are like this in other specializations. So I don’t think my sense of the general value of what’s being done outside the “top journals” is misleading — partial, to be sure, but not misleading. And to echo a concern I raised above, I worry that attitudes like Goodin’s are contributing to the field’s ossification in a time when its survival depends on adaptive growth to a changing environment. Now would be the time to wheel in John Dewey, but I’ll leave it there.