Over the past week, Daily Nous has hosted a discussion of criticisms of the Philosophical Gourmet Report (PGR), a ranking of PhD programs in philosophy. This latest round of criticism began with a post excerpting from “Appearance and Reality in the Philosophical Gourmet Report: Why the Discrepancy Matters to the Profession of Philosophy” by Brian Bruya (Eastern Michigan).  The post generated a lively discussion, and a response from Brian Leiter (Chicago), the creator and, until recently, editor of the PGR. Bruya defended his arguments against Leiter’s response in a follow up guest post here. Following this, David Wallace (Oxford), wrote up a further critique of Bruya’s arguments, which he posted here. Bruya has now written a response to Wallace, which I have posted for him here (more on that, below).

The discussion has largely been interesting and informative. Given the influence of the PGR on the shape of the profession, it is not surprising that a professional journal would take up its methodology (this was not the first time one had done so—-see this). There are legitimate questions about whether the ranking of philosophy departments makes sense, what kinds of rankings (if any) there should be, which factors should count in evaluating programs, how much weight each of those factors should have, who should be doing the ranking, to what uses rankings should be put, how to manage any unintended or undesirable side effects of rankings, and so on. Disagreement on these matters, and mutual criticism of the parties’ approaches to them, is to be expected.

How should such disagreement be handled? We all know the answer to this: subject the arguments to scrutiny and see if they hold up. There has been a good amount of that going on.

How should such disagreement not be handled? We all know the answer to this, too: by impugning the motives and character of the various parties to the dispute. Unfortunately, there has been a good amount of that going on, too.

For example, there have been accusations by Leiter that Bruya’s motivation in writing this piece was to raise the status of the institutions he is associated with. This strikes me as extraordinarily uncharitable. I don’t see why concern for the quality of philosophical inquiry, or the health of the profession, or mere intellectual curiosity, aren’t just as plausible explanations. I know that some people are partial to explanations in terms of self-interest, but as it has been shown—one of my favorite versions of this is by James Rachels in an introductory text—the mere possibility that we can interpret a person’s behavior as self-interested provides scant evidence that the behavior is self-interested.

I would add that the self-interest framing is not only unwarranted, but ineffectual, for it can easily be redirected against the accuser. Leiter’s reaction to Bruya’s criticism, which, in addition to the well-poisoning, included the unjustified accusation of academic fraud, could be interpreted not as a fair evaluation of Bruya’s arguments, but as a self-interested attempt to make Bruya’s criticism of Leiter’s ranking project seem to be not even worth considering, and to deter others from attempting any such criticism, lest they be personally excoriated before Leiter’s readership (as others have, before).

So let’s not go down that road.

(Let me add that I don’t have a horse in this race. My own view is that the PGR has been a mixed bag. In its early years it had the effect of publicizing knowledge of certain people’s opinions of graduate programs, and so was helpful and empowering to many. Yet as its influence grew there were negative effects, too, which have been discussed in detail for years, and as technology has progressed, its “informative” role is less significant. My own preference is for a diversity of evaluative tools, among which may be some version of the PGR—as I’ve said, for example, in a comment on this post here. It is true that Daily Nous has hosted many discussions of criticisms of the PGR, but that is to be expected, given the influence of the PGR on the profession and that Daily Nous is one of the primary online places in which discussions about the philosophy profession take place.)

Some of the PGR’s methods have, commendably, changed over the course of its existence in attempts to improve or counter criticisms. Bruya’s critique, which is of a piece with others that have been made, may provide reasons for further improvements, for fundamental changes, or for the creation of alternatives, or for none of the above. But let’s focus on the arguments.

With that said, Bruya’s response to Wallace is long and detailed—too long for a blog post. I urge readers to look at the whole thing, and everyone is welcome to discuss it here. In it he takes up what I think many believe are the most significant criticisms, including the “egregious explosion of analytic specialties” charge.  He addresses this in a substantive way, but also remarks on the accusations that he was somehow hiding this aspect of his methodology:

Wallace says, “[Bruya’s classification method] appears nowhere in the main paper, which is entirely silent on the reclassification” (p. 1). This is not correct.  I begin looking closely at area classification in section E, on p. 669.  I immediately provide a footnote (number 15) referring the reader to Appendix 2, where my methods are explained in full. I originally had it all, beginning right there on the same page, in footnote 15, but the journal said it was too long for a footnote, so we decided to move it to an appendix.  The case is the same with the other appendix, by the way.  To immediately refer the reader from a footnote to an explanation in the appendix seems perfectly reasonable to me.  It is not hidden if I point the reader directly to it.

In his concluding section, Bruya asks that his critics consider some related questions. I share these below, should readers wish to discuss them:

[q1.] Is it true that some philosophy Ph.D. programs in the U.S. use the PGR to build their programs?

By this, I mean, does anyone in the hiring process ever ask “How will hiring this person affect our rank in the PGR?” (Alternatively, a program could use a PGR rank to substantiate its status in the profession to administrators or aspire to do so.)  If the answer to this question is: “No, no significant number of programs use the PGR to build their programs,” then my article is of little value, and there is no point in even critiquing it, except to say that it relies on this one flawed assumption.  If, however, the answer is “yes,” then the next question must be asked:

[q2.] Is it true that hiring in any one specialty over other specialties can affect one’s overall ranking, all else being equal? I show decisively, and others have also shown, that it does. 

If both 1 and 2 are true, then there must be structural flaws in the PGR which will lead to artificial and damaging imbalances in the field over the long run, as programs pursue specialists in fields that will raise their PGR rank over specialists in fields that won’t, or won’t do it as well.  [Perhaps someone who works on game theory could try to model this process to see where it will lead.] I ask in my article and try to answer the question of what those structural flaws are.  The existence and deleterious effects of these flaws shout out for a response to a third question:

[q3.] How can the PGR be reformed to be more inclusive?

All programs that want to raise or maintain their rank in the PGR have a very strong incentive to hire in fields already well-represented in the PGR and a very strong negative incentive not to take risks on marginalized fields.  Over time, this will mean that marginalized fields will become more marginalized until disappearing altogether.  This is exactly what is happening to Chinese philosophy.  Fifteen years ago, there were specialists at Berkeley, Stanford, and Michigan.  Instead of that number growing in other elite schools as multiculturalism and globalization and diversity have been spreading steadily through universities, now only one half-timer remains at Berkeley.  The situation for Chinese philosophy is not unique.  All non-Western philosophies are largely non-existent in PGR-ranked philosophy Ph.D. programs.  There are many other marginalized fields as well (as I note in my article, and as others have noted) that face the same fate over time.  Maybe all the blame can’t be put at the feet of the PGR, but if any of the blame can, it is a reason in favor of reforming the PGR.  In my article, I offer five suggestions for reform.  Instead of focusing narrow critiques on minor parts of my article, let’s open a discussion on all possible methods of reform.

He also adds:

1. I appreciate the attention the article is receiving and hope that it contributes to rectifying the situation, but I can’t keep up the pace of responding to blog posts. I suggest that anyone who has further criticisms of my critique write them up and get them published. Maybe we can start a little cottage industry examining the pros and cons of the PGR.

2. On that note, I would like to draw attention to another peer-reviewed critique of the PGR: Jennifer Saul’s excellent “Ranking Exercises in Philosophy and Implicit Bias” (Journal of Social Philosophy, v. 43, n. 3). Here is a thought experiment that she puts forward:

“The practical effects of this [implicit bias in the PGR] are even more disturbing [than the under-rating of women philosophers in the PGR]. And here, it is important to note that the Gourmet Report is only intended to serve as a starting point for those considering graduate school.  Despite this limited intention, the report has come to play some rather influential unintended roles: most significantly, perhaps, as a guide for university administrators, who are known to demand that new hires raise a department’s ranking and to criticize departments whose ranking declines.

“To begin to appreciate the damaging effects of all of this, imagine that you are on the hiring committee for a department that has been told to raise its ranking in the Gourmet Report. You now find yourself faced with two candidates who seem to you equally good as researchers, as teachers, as colleagues, and so on. One candidate is a woman and one candidate is a man. (We will ignore the fact that implicit bias is probably affecting your judgment that the two candidates are of equal quality.) What should you do? Well, if you are familiar with the literature that I have just discussed, and if you understand the working of the Gourmet Report, the answer is clear: you should hire the man. If the research of these two candidates is truly of equal quality, it is likely that the man’s work will be judged to be of higher quality by those filling out Gourmet Report surveys. And when the survey participants are ranking whole departments and his name is one among many dimly remembered names, it is likely to be judged more famous than her name would be. The instruction from your dean, then, amounts to an instruction to discriminate against members of stigmatized groups.”(268)

I was inspired to publish my piece after reading Saul’s article.  Her piece also offers a critique and valuable potential solutions.  Implicit bias is another angle from which the PGR board of advisors should look at reforming the PGR.  Take measures to reduce its influence as much as possible.  Saul doesn’t see anything being done about reforming the PGR in substantial ways, and neither do I, but both of us have suggestions on how to do it.  Maybe someone of influence can open a conversation to discuss them.

3. There is one very large potential sticking point to the prospect of reforming the PGR. Let’s say there is a discussion of solutions to reforming the PGR, but none can be agreed on as workable, or none actually gets implemented. Then what?  Then, it should be scrapped altogether for the damage it is doing to the profession.  Gregory Wheeler argues that the PGR is not reformable at all and so recommends that it be abolished now.

4. I would like to draw your attention also to the many other critiques of the PGR on sites across the blogosphere, as listed in my bibliography. These also were a great inspiration to me and were what prompted Leiter to defend the PGR on his blog in the first place, defenses that I take issue with in my article. Saul and I are not the only ones who have noticed and detailed significant problems in the PGR.

5. I’ve seen comments saying something to the effect “If you don’t like the PGR, create your own equivalent.” This is like telling a member of an excluded minority, “If you don’t like our exclusive white, heterosexual country club, go create your own.” No attempt to actually recognize the systemic roots of the issue, let alone rectify the situation.  Actually, the analogy of the country club is not off the mark in the sense that country clubs function (or used to function) as gatekeepers to the upper echelons of society.  The PGR functions this way, too.  This is why its exclusivity is so problematic.  On a closely related topic, a colleague in the social sciences suggested to me as I was writing this article that I should refer to the literature on gatekeepers in academia.  I didn’t do that, but perhaps that is another angle that someone would like to take up and pursue for a future article.

6. I would also like to direct readers’ attention to the reason I wrote this article in the first place. My “Appearance and Reality” article started out as an appendix to the article “The Tacit Rejection of Multiculturalism in Philosophy Ph.D. programs” in order to support my claims there before eventually outgrowing that role. I would love to see a blog discussion of the arguments in the “Tacit Rejection” article, especially my argument for the value of pluralism.  The “Tacit Rejection” article was also originally intended as an appendix—to the book The Philosophical Challenge from China (MIT Press, 2015)—yes, I’m shamelessly plugging my own work, but it is relevant.  I would love to also see some discussion of the Introduction to that book and even of its contents and their relation to the prospect of widening the scope of analytic philosophy into non-Western resources.

7. Some people seem to look at the field of philosophy as if it were an economy, with rules of supply and demand governing hiring decisions, and various programs creating niches for themselves within this economy. This is an attractive way to view the field because it allows for autonomy within all programs under the assumption that when all these autonomous programs pursue their own self-interest, the entire profession will flourish.  When the PGR is introduced into this economy, however, it creates a situation analogous to the classic tragedy of the commons.  Suddenly there is a limited resource that everyone pursues at the expense of the greater good.  The purpose of my article is to turn our attention to this greater good and see how the PGR can be reformed in order to better serve it.

8. In section 5 above, a hypothetical department is discussed, one that has a high concentration of scholars in specialties that the PGR does not recognize. How plausible is the existence of such a department, and could such a department, if found, act as a test case for inclusiveness in the PGR? The PGR purports to represent the entire profession of philosophy. One of the main underlying assumptions of the PGR, in its role as a useful tool for prospective graduate students, is that there is a strong correlation between the quality of faculty in a program and the job placement rate when graduates come out of that program.  If we can find a program that has a high job placement rate but does not make the PGR rankings, that would suggest a blind spot in the PGR evaluator pool.  I know of just such a program: the University of Hawai’i.  According to data on the UH philosophy website, UH’s initial tenure-track placement rate for graduates since 2004 is 49%.  According to a report put out by PhilosophyNews (using data from the year 2000), the figure of 49% places UH exactly even with Princeton University’s philosophy program, and above the University of Chicago, University of Michigan, MIT, Carnegie Mellon, and Stanford, all well-ranked PGR programs.  [The initial tenure-track placement statistic does not take into account other placements such as post-docs, visiting positions, lectureships, non-academic jobs, later placement, or the quality of the school, but for many graduates faced with adjuncting or outright unemployment, placement into a tenure-track position is the gold standard].  So if UH is placing graduates into tenure-track positions at a rate that exceeds most PGR-ranked programs, by the underlying rationale of the PGR (helping students choose programs that will get them jobs across the entire spectrum of philosophy), then UH should rank also.

The fact that UH doesn’t register even a blip on the overall PGR rankings points to a blind spot in the evaluator pool.  What is the pool lacking that UH has?  Here is a list of the number of UH faculty working in various specialties, drawing from the specific listings on their faculty webpage and allowing for multiple specialties:

Aesthetics 2
American 2
Ancient Greek 1
Buddhist 1
Chinese 3
Comparative 4
Confucianism 1
Continental 1
Daoism 1
Environmental 1
Epistemology 2
Ethics 3
Feminism 2
Hermeneutics 1
Historical ontology 1
Indian 2
Islamic 1
Japanese  2
Mathematical logic 1
Metaphysics 2
Modern 1
Moral Psychology 1
Neo-Confucianism 1
Phenomenology 1
Philosophy for Children 1
Philosophy of Language 1
Philosophy of Law 2
Philosophy of Religion 1
Political 2
Process philosophy 1
Renaissance 1

Since most UH graduates are placed into positions in non-Western fields, the blind spot is obvious—specialties in non-Western philosophy, which are almost entirely absent in the PGR evaluator pool.

I happen to be a graduate of UH and so am not entirely unbiased on this issue, but the facts are the facts, and my association with UH doesn’t change its value as a test case for assessing the inclusiveness of the PGR, using the PGR’s own standards.  Remember, the PGR does purport to represent the entire profession, not just part of it.  Surely there are other test cases as well, and concerned scholars, like David Wallace and the PGR’s own board, should look into these to see how the PGR can be improved and live up to its own aspirations for providing reliable information to all prospective philosophy graduate students.

(I know, making this claim opens me up to all kinds of sour grapes arguments: “Bruya’s test case is a laughing stock: If his alma mater doesn’t make the ranking, then the ranking must be flawed.  That’s what they all say!”  But it is a good test case—for the reasons given.)

9. Finally, the PGR should look to the U.S. Fulbright program as a model of inclusiveness. Fulbright is one of the most prestigious fellowship programs in the U.S., but it doesn’t draw it’s awardees or its evaluators only from the most elite universities.  It draws from community colleges right up through Ivy League schools, always attempting to retain a balance.  If the PGR really aspires to represent the entire profession of philosophy by providing generalizable conclusions about what philosophers think about faculty quality in Ph.D. programs, or generalizable information about all philosophy Ph.D. programs that will place students into jobs, or generalizable information about job placement prospects into schools, then it should strive for a similar kind of inclusiveness.

Bruya’s entire reply is here.