How Often Are Philosophy Articles Actually Cited? Encouraging News (guest post)
In the following guest post, Eric Schwitzgebel (UC Riverside) recounts what he found when, prompted by claims about how infrequently academic philosophy articles are cited, he looked at the citation rates of articles published in a few journals a decade ago.
(A version of this post first appeared at his blog, The Splintered Mind.)
How Often Are Philosophy Articles Actually Cited? Encouraging News
by Eric Schwitzgebel
You hear terrible things. You hear, for instance, that over 50% of philosophy articles are entirely uncited. You hear that the average philosophy article is cited less than 5 times. People will sometimes say things like the “median number of readers for a philosophy article is 1” or “there’s not much difference between publishing a paper and throwing it away“. Both of the last two comments appear in a recent Twitter thread launched by Helen De Cruz. It was reading this thread that inspired me to do the analyses I’ll share here.
Generally my reaction to analyses and comments of this sort is to think that they are considerably underestimating how often philosophy articles are actually cited. The “big data” interdisciplinary analyses often use methods that are a terrible fit for philosophy, such as looking at citations in the past two years. (Often in philosophy it takes two years or more to write and publish an article.) Also, I wonder what counts as an “article”. If we’re including two-page book reviews, it’s little wonder that they’d be little cited, and similarly if we’re including publications in predatory or obscure journals. What would be more interesting to know—and what I think most people in this discussion really care about—is how frequently cited are full-length research articles in “respectable” mainstream philosophy journals?
Before you look at my analyses, any guesses?
Method
I selected six representative general philosophy journals for analysis: two “top ranked” journals (Philosophical Review and Noûs), two mid-ranked journals (Canadian Journal of Philosophy and Pacific Philosophical Quarterly), and two unranked but reputable journals (Philosophia and Southern Journal of Philosophy). I then downloaded the entire table of contents of these journals from the year 2012—giving a ten-year citation window—and excluded anything that wasn’t an ordinary full-length research article (e.g., book reviews, editors’ introductions, symposium proceedings). From each journal, I randomly selected 15 articles and noted their total number of citations in Google Scholar.
Results
Although this is not a large database, the results from this carefully designed search are, I think, striking.
Overall, the mean article was cited 31 times, and the median number of citations was 20. For the elite journals the mean was 50 and the median was 36; for the mid-ranked journals the mean was 25 and the median was 8; for the unranked journals, the mean was 17 and the median was 12. Only three of the 90 articles (3%) were cited zero times.
Here’s a more specific breakdown, journal by journal. Of course with only 15 articles per journal, there will be lots of noise in the numbers when considered at this fine a grain.
Histogram of total citations by article for articles published in 2012 from six representative general philosophy journals.
Conclusions
I draw the following general conclusions:
(1) Very few articles published in mainstream general philosophy journals remain entirely uncited. My estimate of 3% might be off, due to the smallish sample size and the particular journals selected; but it’s highly unlikely that non-citation rates of mainstream general philosophy articles are anything near 50%.
(2) The majority of articles are cited often enough to be “in the conversation”. While an outsider to academia might not think that ten citations is much impact, I submit that a more appropriate perspective is this: If your article is cited at least ten times, then it is having an impact on other specialists in your subfield. Your article isn’t falling into the void. It is part of the conversation, and other scholars are reacting to it. A journal article needn’t have a huge influence outside its subarea to be successful. The beauty of academic journals is that they host technical pieces that often can only be appreciated by a few dozen specialists. If academia is worthwhile, then continuing those specialists’ conversations is worthwhile, and high citation rates for technical pieces should not be expected. (I intend this ten citation criterion as an approximately sufficient condition for impact, not a necessary condition.)
(3) Articles in elite journals are cited only about three times as often as articles in unranked but reputable journals. To me, this was the most surprising result, and part of me wonders whether we’d see different results if we looked at different years or different unranked but reputable journals. Philosophical Review is much more prestigious in the eyes of the typical mainstream academic philosopher than is the typical unranked but reputable journal. I find this result encouraging, especially given the tiny acceptance rates at the elite journals (Philosophical Review reports accepting about twelve articles per year out of 600 submitted). Publishing in unranked journals is not shouting into the void. It is not defeat. The dynamic is less jackpot-or-nothing than I would have expected. You don’t have to get into a top-ten journal to have an impact. Philosophers do regularly read and cite articles from the less prestigious journals.
(4) Citation skew is much less extreme than it could be. Another respect in which philosophy citation practices are not jackpot-or-nothing is revealed by the smallish differences between the means and medians. Of course the means are higher: there’s right skew in the data, a tendency for high-end outliers to pull up the mean. But it’s not like most articles are cited 0-5 times and a few are a cited hundreds of times. Authors can reasonably expect that a decent article in a decent journal will have at least a moderate impact on their subfield.
Related: “Philosophy Has High Rate of Uncited Publications“, “Citation Patterns Across Journals“, “Philosophers Don’t Read and Cite Enough“, “Citation Problems in Philosophy—and Some Fixes“, “Citation Rankings of Philosophers Based on Scopus Data“. See also, “Where Are They Now? The Healy 2100” by Jonathan Weisberg at his site.
For what it’s worth, even apart from books, Google Scholar doesn’t catch all citations (e.g. a few of my papers have been cited by papers that Google Scholar knows about, but Google Scholar does not realize that those papers cite my papers). So that’s another reason to bump the numbers up a little.
Regarding (2), surely this depends on whether citations actually imply any serious engagement with the work? Citations can reflect all sorts of reasons (“I needed to support this claim”, “this person might be a reviewer”, “a reviewer asked me to include this”). I cannot find it now, but some (large-scale) analyses seemed to suggest that most citations do not evaluate the cited work. My own, non-philosophical work regularly gets cited in ways that suggest that the authors have not actually read the paper, far less understood it. I suspect that philosophy suffers less from this than other, especially empirical disciplines (norms appear to be closer to citing only work with which one actually engages). But interpreting citations as conversation might be too big a jump without a look at their context.
This is a good point. There is also something like the opposite case: articles which are read and have influence, but are not cited.
It would be interesting to compare the above to other quantifiable thingies, like download frequency for the same publications, blog or social media mentions, germinal conference presentations etc. What else could stand in for contributing to the conversation?
Or what about reference referencing, where you read a paper that mentions something cool from a different paper, go read the original and then cite that instead of the paper that led you to it. In that way some papers which are read and not cited might have significant influence (as a publicity publication) even when they don’t have many citations.
And this might count in favour of the humble book review. How often are book reviews responsible for a book’s becoming well known and frequently cited and thereby being an influential though uncited publication?
If the mean article receives 62 citations in its lifetime, that sounds like it should suggest that the mean article cites 62 articles? I suppose not exactly, since some articles are cited by reviews and other things that aren’t in this study (and presumably don’t get cited much themselves). And there’s some amount of philosophy articles citing things outside the discipline and vice versa.
But this still sounds like a large number to me!
Scholar sometimes captures citations from dissertations, unpublished manuscripts posted on scholarly cites, and articles in journals that aren’t well known, as well as some (I suspect smallish) fraction of scholarly books. Also, it’s certainly possible that there’s most citation of philosophy by other disciplines than vice versa, though I don’t have systematic data on this.
As a quick example, take Simpson’s “What Is Trust”, one of the more highly cited articles in my sample. The most recent 10 citations are from: one book, one MA thesis, one PhD thesis, and the following seven journals: International Journal of Philosophical Studies, Religious Studies, Journal of Applied Philosophy, Philosophy & Technology, Technology in Society, Behaviour and Information Technology, Information Technology. The selection of journals suggests that the subliterature on information technology ethics has picked up on this article.
Or taking one that is cited at about mean rates, consider Schulte’s “How frogs see the world”. The most recent ten citations appear to be two books, an apparently unpublished manuscript, and the journals Mind & Language, Philosophical Quarterly (twice), Harvard Review of Philosophy, Revista de Filosofia Moderna, Synthese, and Philosophical Studies.
Sheesh, sorry about the typos!
This seems to scattershot to draw any real conclusions. I wonder if someone couldn’t use Google Scholar to pull data more fully from all philosophy journals to take a broader look at citation counts? It seems like any approaches are going to miss a lot of data and people (as noted, GS doesn’t count citations in books but it counts books that are cited).
A lot of people don’t have GS profiles so looking only at profiles would seem like a mistake. The citation counts that individual journals use on their own pages often differ wildly from GS counts. I’ve also noticed, looking at my own GS citations, that GS counts a lot of things as citations that I wouldn’t normally count (undergraduate theses , etc.).
I actually used to think that I was doing okay in terms of citations but if Eric is right then actually NONE of my articles are anywhere near the median which, to be honest, I find somewhat depressing. In fact, the GS profiles of faculty in my department are *radically* underperforming if the median # of citations is in the 40s (or, as Eric suggests, maybe even in the 60s). That’s actually kind of depressing but also strikes me as counterintuitive unless we have better, fuller, data to support it.
Maybe it would be useful if people could provide some of their own (or their department’s) data in this thread? How many articles do you have in your GS profile with 0-5, 6-10, 11-20, 21-30, 30+ citations?
ME (10 years out from the PhD):
0-5 12
6-10 4
11-20 1
21-30 2
30+ 1
as noted, GS doesn’t count citations in books
I’m not sure I understand the claim here. It sounds like it’s being claimed that, if your paper is cited in a book, then Google scholar won’t count it. I do know of some cases where citations of my work in books haven’t shown up in Google Scholar, but I also have a good number of citations of my work in books that do show up there. So, if that’s the claim, it’s too strong. If something else is the claim, I’d be intrested to hear what it is.
Eric’s conjecture of a median of 40 was for lifetime citations. The median after a decade was 20. Since you’re only a decade post-phd, and so presumably published most of those articles significantly less than ten years ago, it’s not clear that your numbers are wildly out of line with that stat.
This is why department-level data is important too. Looking at the most senior people in my department (a department I would consider at the very least “average” if not above average), Eric’s numbers just don’t bear out. My conjecture is that Eric’s conjecture is overstimating citations in two different ways:
I think both of these are likely to *dramatically* inflate counts of what an average philosopher publishing in an average journal is likely to see in terms of citations and both are likely due to Eric’s experience in mostly R-1 environments. Remember that the average philosopehr is not working at an R-1 or even a SLAC.
I’ve been on a lot of search committees and I’m used to seeing people list publications in places like the “Journal of World Philosophies” or book chapters for collections from all over the place. The critique here, in other words, is that nothing about Eric’s search for journals or citations is really about the average philosopher and so I don’t think that his conjectures really do much to undermine the claim that the average journal is cited 0-3 times.
It might be better to say that Eric is showing us that folks who work among the most rearch competitive top 20% of philosophy jobs are cited more often than most but…I think we already knew that?
Sorry if the results seem depressing, CG! But as grymes suggests, at only ten years out, most of your articles have probably only begun to gather citations. Some of my most-cited articles got few citations in their first several years. For example, my “Phenomenal, Dispositional Account of Belief”, currently at 433 citations, got 0-5 citations per year in Google Scholar from its publication in 2002 through 2008.
You’re presumably right about lower rates of citation in less well-known journals than the strong but un-Leiter-ranked Southern Journal of Philosophy and Philosophia. Now I’m curious. We could of course use the same method to look at the results from a sample of such journals.
Hey Eric! Thanks for replying and sorry for taking so long to get back to you. I think that what got me about your analysis was that it opened with one sort of problem:
>You hear, for instance, that over 50% of philosophy articles are entirely uncited. You hear that the average philosophy article is cited less than 5 times. People will sometimes say things like the “median number of readers for a philosophy article is 1” or “there’s not much difference between publishing a paper and throwing it away“.
But it ends up answering what I take to be a totally different question (something like ‘how many times is the average philosophy article cited if it’s published among the top 20% of philosophy journals’). Your answer (a few dozen lifetime citations) doesn’t really do anything to address the question of the “average” article.
We’d need to operationalize that first and there I don’t think it’s good to go with prestige rankings at all. It’s unlikely that the average article is published in the Sourthern Journal of Philosophy just as it’s unlikely that the average article is an undergraduate senior thesis. That leaves a lot of room about what average articles really are: do we need to find the median philosophy job and see where those people tend to publish? If we go that route we might find that the median job is an NTT position at an unranked university who publishes something in a regional journal. If we go by something like median journal by output then we might find that the median journal is a predatory one (there are so many of them!). Do we only count articles in English?
But if we instead choose to go by some gut operationalizing, as I think you kind of did here, then we don’t really answer the original question. It seems that, in order to make progress on this at all, we’d need to really start with the definition of the “average” or “median” journal, article, or philosopher. Only then can we figure out how often citations happen.
The methodology used in this exercise strikes me as questionable and I’d caution against drawing any conclusions from it. It would be interesting to randomly select 15 articles from each of two randomly selected top, mid, unranked journals and see how much the mean and median citation numbers fluctuate. I looked at the mean citations of all 2012 CJP articles on Google Scholar (Using a “source:”Canadian journal of philosophy” search w date range set to 2012). The average number of citations per article seems well below the mid-ranked mean of 25 citations stated here (my quick calculation returns a mean of 14.5 citations per article, excluding the Hume Conference issue which would only drag down that number).
In my experience Google scholar is a big net undercounter (often missing books, as well as theses and articles in languages other than English) but there is one respect in which it overcounts and this may be skewing Eric’s figures upwards. If your article is listed on PhilPapers, then Google Scholar counts the thematically-linked articles as citations despite the fact that many of them don’t mention you at all. Did Eric discount the these Google-Scholar generated pseudo-citations? If not, the citation averages may be looking healthier than they really are.
One of the biggest challenge of philosophy is the fact that it thrives in individual thought system, instead of professional methodology.
From the inception of philosophy, individual philosophers held different views on different issues resulting to different schools of thoughts,even within the schools of thoughts there are still diversity of positions. In fact, there many ideas as there are many thoughts in philosophy. This particularly constitutes a lot of challenges because the writer’s submission might not necessarily appeal to the editors of the journals,just as it might not likely appeal to other coming writers in citation,and any detailed changes might make the essay to lose the original meaning as initially conceived by the author.
So it is understandable if philosophy records low citation of writers than other professions, because hardly would any other profession record the kind of diversity of philosophy.
Igweolisa Sunday Nebeolisa-Igwe is a Human development philosopher and propounder of Reminiscencing-transcendency Theory/RTT (Nebeolism-Igweism)