Criticism of the Philosophical Gourmet Report

Criticism of the Philosophical Gourmet Report


Over the past week, Daily Nous has hosted a discussion of criticisms of the Philosophical Gourmet Report (PGR), a ranking of PhD programs in philosophy. This latest round of criticism began with a post excerpting from “Appearance and Reality in the Philosophical Gourmet Report: Why the Discrepancy Matters to the Profession of Philosophy” by Brian Bruya (Eastern Michigan).  The post generated a lively discussion, and a response from Brian Leiter (Chicago), the creator and, until recently, editor of the PGR. Bruya defended his arguments against Leiter’s response in a follow up guest post here. Following this, David Wallace (Oxford), wrote up a further critique of Bruya’s arguments, which he posted here. Bruya has now written a response to Wallace, which I have posted for him here (more on that, below).

The discussion has largely been interesting and informative. Given the influence of the PGR on the shape of the profession, it is not surprising that a professional journal would take up its methodology (this was not the first time one had done so—-see this). There are legitimate questions about whether the ranking of philosophy departments makes sense, what kinds of rankings (if any) there should be, which factors should count in evaluating programs, how much weight each of those factors should have, who should be doing the ranking, to what uses rankings should be put, how to manage any unintended or undesirable side effects of rankings, and so on. Disagreement on these matters, and mutual criticism of the parties’ approaches to them, is to be expected.

How should such disagreement be handled? We all know the answer to this: subject the arguments to scrutiny and see if they hold up. There has been a good amount of that going on.

How should such disagreement not be handled? We all know the answer to this, too: by impugning the motives and character of the various parties to the dispute. Unfortunately, there has been a good amount of that going on, too.

For example, there have been accusations by Leiter that Bruya’s motivation in writing this piece was to raise the status of the institutions he is associated with. This strikes me as extraordinarily uncharitable. I don’t see why concern for the quality of philosophical inquiry, or the health of the profession, or mere intellectual curiosity, aren’t just as plausible explanations. I know that some people are partial to explanations in terms of self-interest, but as it has been shown—one of my favorite versions of this is by James Rachels in an introductory text—the mere possibility that we can interpret a person’s behavior as self-interested provides scant evidence that the behavior is self-interested.

I would add that the self-interest framing is not only unwarranted, but ineffectual, for it can easily be redirected against the accuser. Leiter’s reaction to Bruya’s criticism, which, in addition to the well-poisoning, included the unjustified accusation of academic fraud, could be interpreted not as a fair evaluation of Bruya’s arguments, but as a self-interested attempt to make Bruya’s criticism of Leiter’s ranking project seem to be not even worth considering, and to deter others from attempting any such criticism, lest they be personally excoriated before Leiter’s readership (as others have, before).

So let’s not go down that road.

(Let me add that I don’t have a horse in this race. My own view is that the PGR has been a mixed bag. In its early years it had the effect of publicizing knowledge of certain people’s opinions of graduate programs, and so was helpful and empowering to many. Yet as its influence grew there were negative effects, too, which have been discussed in detail for years, and as technology has progressed, its “informative” role is less significant. My own preference is for a diversity of evaluative tools, among which may be some version of the PGR—as I’ve said, for example, in a comment on this post here. It is true that Daily Nous has hosted many discussions of criticisms of the PGR, but that is to be expected, given the influence of the PGR on the profession and that Daily Nous is one of the primary online places in which discussions about the philosophy profession take place.)

Some of the PGR’s methods have, commendably, changed over the course of its existence in attempts to improve or counter criticisms. Bruya’s critique, which is of a piece with others that have been made, may provide reasons for further improvements, for fundamental changes, or for the creation of alternatives, or for none of the above. But let’s focus on the arguments.

With that said, Bruya’s response to Wallace is long and detailed—too long for a blog post. I urge readers to look at the whole thing, and everyone is welcome to discuss it here. In it he takes up what I think many believe are the most significant criticisms, including the “egregious explosion of analytic specialties” charge.  He addresses this in a substantive way, but also remarks on the accusations that he was somehow hiding this aspect of his methodology:

Wallace says, “[Bruya’s classification method] appears nowhere in the main paper, which is entirely silent on the reclassification” (p. 1). This is not correct.  I begin looking closely at area classification in section E, on p. 669.  I immediately provide a footnote (number 15) referring the reader to Appendix 2, where my methods are explained in full. I originally had it all, beginning right there on the same page, in footnote 15, but the journal said it was too long for a footnote, so we decided to move it to an appendix.  The case is the same with the other appendix, by the way.  To immediately refer the reader from a footnote to an explanation in the appendix seems perfectly reasonable to me.  It is not hidden if I point the reader directly to it.

In his concluding section, Bruya asks that his critics consider some related questions. I share these below, should readers wish to discuss them:

[q1.] Is it true that some philosophy Ph.D. programs in the U.S. use the PGR to build their programs?

By this, I mean, does anyone in the hiring process ever ask “How will hiring this person affect our rank in the PGR?” (Alternatively, a program could use a PGR rank to substantiate its status in the profession to administrators or aspire to do so.)  If the answer to this question is: “No, no significant number of programs use the PGR to build their programs,” then my article is of little value, and there is no point in even critiquing it, except to say that it relies on this one flawed assumption.  If, however, the answer is “yes,” then the next question must be asked:

[q2.] Is it true that hiring in any one specialty over other specialties can affect one’s overall ranking, all else being equal? I show decisively, and others have also shown, that it does. 

If both 1 and 2 are true, then there must be structural flaws in the PGR which will lead to artificial and damaging imbalances in the field over the long run, as programs pursue specialists in fields that will raise their PGR rank over specialists in fields that won’t, or won’t do it as well.  [Perhaps someone who works on game theory could try to model this process to see where it will lead.] I ask in my article and try to answer the question of what those structural flaws are.  The existence and deleterious effects of these flaws shout out for a response to a third question:

[q3.] How can the PGR be reformed to be more inclusive?

All programs that want to raise or maintain their rank in the PGR have a very strong incentive to hire in fields already well-represented in the PGR and a very strong negative incentive not to take risks on marginalized fields.  Over time, this will mean that marginalized fields will become more marginalized until disappearing altogether.  This is exactly what is happening to Chinese philosophy.  Fifteen years ago, there were specialists at Berkeley, Stanford, and Michigan.  Instead of that number growing in other elite schools as multiculturalism and globalization and diversity have been spreading steadily through universities, now only one half-timer remains at Berkeley.  The situation for Chinese philosophy is not unique.  All non-Western philosophies are largely non-existent in PGR-ranked philosophy Ph.D. programs.  There are many other marginalized fields as well (as I note in my article, and as others have noted) that face the same fate over time.  Maybe all the blame can’t be put at the feet of the PGR, but if any of the blame can, it is a reason in favor of reforming the PGR.  In my article, I offer five suggestions for reform.  Instead of focusing narrow critiques on minor parts of my article, let’s open a discussion on all possible methods of reform.

He also adds:

  1. I appreciate the attention the article is receiving and hope that it contributes to rectifying the situation, but I can’t keep up the pace of responding to blog posts. I suggest that anyone who has further criticisms of my critique write them up and get them published. Maybe we can start a little cottage industry examining the pros and cons of the PGR.
  1. On that note, I would like to draw attention to another peer-reviewed critique of the PGR: Jennifer Saul’s excellent “Ranking Exercises in Philosophy and Implicit Bias” (Journal of Social Philosophy, v. 43, n. 3). Here is a thought experiment that she puts forward:

“The practical effects of this [implicit bias in the PGR] are even more disturbing [than the under-rating of women philosophers in the PGR]. And here, it is important to note that the Gourmet Report is only intended to serve as a starting point for those considering graduate school.  Despite this limited intention, the report has come to play some rather influential unintended roles: most significantly, perhaps, as a guide for university administrators, who are known to demand that new hires raise a department’s ranking and to criticize departments whose ranking declines.

“To begin to appreciate the damaging effects of all of this, imagine that you are on the hiring committee for a department that has been told to raise its ranking in the Gourmet Report. You now find yourself faced with two candidates who seem to you equally good as researchers, as teachers, as colleagues, and so on. One candidate is a woman and one candidate is a man. (We will ignore the fact that implicit bias is probably affecting your judgment that the two candidates are of equal quality.) What should you do? Well, if you are familiar with the literature that I have just discussed, and if you understand the working of the Gourmet Report, the answer is clear: you should hire the man. If the research of these two candidates is truly of equal quality, it is likely that the man’s work will be judged to be of higher quality by those filling out Gourmet Report surveys. And when the survey participants are ranking whole departments and his name is one among many dimly remembered names, it is likely to be judged more famous than her name would be. The instruction from your dean, then, amounts to an instruction to discriminate against members of stigmatized groups.”(268)

I was inspired to publish my piece after reading Saul’s article.  Her piece also offers a critique and valuable potential solutions.  Implicit bias is another angle from which the PGR board of advisors should look at reforming the PGR.  Take measures to reduce its influence as much as possible.  Saul doesn’t see anything being done about reforming the PGR in substantial ways, and neither do I, but both of us have suggestions on how to do it.  Maybe someone of influence can open a conversation to discuss them.

  1. There is one very large potential sticking point to the prospect of reforming the PGR. Let’s say there is a discussion of solutions to reforming the PGR, but none can be agreed on as workable, or none actually gets implemented. Then what?  Then, it should be scrapped altogether for the damage it is doing to the profession.  Gregory Wheeler argues that the PGR is not reformable at all and so recommends that it be abolished now.
  1. I would like to draw your attention also to the many other critiques of the PGR on sites across the blogosphere, as listed in my bibliography. These also were a great inspiration to me and were what prompted Leiter to defend the PGR on his blog in the first place, defenses that I take issue with in my article. Saul and I are not the only ones who have noticed and detailed significant problems in the PGR.
  1. I’ve seen comments saying something to the effect “If you don’t like the PGR, create your own equivalent.” This is like telling a member of an excluded minority, “If you don’t like our exclusive white, heterosexual country club, go create your own.” No attempt to actually recognize the systemic roots of the issue, let alone rectify the situation.  Actually, the analogy of the country club is not off the mark in the sense that country clubs function (or used to function) as gatekeepers to the upper echelons of society.  The PGR functions this way, too.  This is why its exclusivity is so problematic.  On a closely related topic, a colleague in the social sciences suggested to me as I was writing this article that I should refer to the literature on gatekeepers in academia.  I didn’t do that, but perhaps that is another angle that someone would like to take up and pursue for a future article.
  1. I would also like to direct readers’ attention to the reason I wrote this article in the first place. My “Appearance and Reality” article started out as an appendix to the article “The Tacit Rejection of Multiculturalism in Philosophy Ph.D. programs” in order to support my claims there before eventually outgrowing that role. I would love to see a blog discussion of the arguments in the “Tacit Rejection” article, especially my argument for the value of pluralism.  The “Tacit Rejection” article was also originally intended as an appendix—to the book The Philosophical Challenge from China (MIT Press, 2015)—yes, I’m shamelessly plugging my own work, but it is relevant.  I would love to also see some discussion of the Introduction to that book and even of its contents and their relation to the prospect of widening the scope of analytic philosophy into non-Western resources.
  1. Some people seem to look at the field of philosophy as if it were an economy, with rules of supply and demand governing hiring decisions, and various programs creating niches for themselves within this economy. This is an attractive way to view the field because it allows for autonomy within all programs under the assumption that when all these autonomous programs pursue their own self-interest, the entire profession will flourish.  When the PGR is introduced into this economy, however, it creates a situation analogous to the classic tragedy of the commons.  Suddenly there is a limited resource that everyone pursues at the expense of the greater good.  The purpose of my article is to turn our attention to this greater good and see how the PGR can be reformed in order to better serve it.
  1. In section 5 above, a hypothetical department is discussed, one that has a high concentration of scholars in specialties that the PGR does not recognize. How plausible is the existence of such a department, and could such a department, if found, act as a test case for inclusiveness in the PGR? The PGR purports to represent the entire profession of philosophy. One of the main underlying assumptions of the PGR, in its role as a useful tool for prospective graduate students, is that there is a strong correlation between the quality of faculty in a program and the job placement rate when graduates come out of that program.  If we can find a program that has a high job placement rate but does not make the PGR rankings, that would suggest a blind spot in the PGR evaluator pool.  I know of just such a program: the University of Hawai’i.  According to data on the UH philosophy website, UH’s initial tenure-track placement rate for graduates since 2004 is 49%.  According to a report put out by PhilosophyNews (using data from the year 2000), the figure of 49% places UH exactly even with Princeton University’s philosophy program, and above the University of Chicago, University of Michigan, MIT, Carnegie Mellon, and Stanford, all well-ranked PGR programs.  [The initial tenure-track placement statistic does not take into account other placements such as post-docs, visiting positions, lectureships, non-academic jobs, later placement, or the quality of the school, but for many graduates faced with adjuncting or outright unemployment, placement into a tenure-track position is the gold standard].  So if UH is placing graduates into tenure-track positions at a rate that exceeds most PGR-ranked programs, by the underlying rationale of the PGR (helping students choose programs that will get them jobs across the entire spectrum of philosophy), then UH should rank also.

The fact that UH doesn’t register even a blip on the overall PGR rankings points to a blind spot in the evaluator pool.  What is the pool lacking that UH has?  Here is a list of the number of UH faculty working in various specialties, drawing from the specific listings on their faculty webpage and allowing for multiple specialties:

Aesthetics 2
American 2
Ancient Greek 1
Buddhist 1
Chinese 3
Comparative 4
Confucianism 1
Continental 1
Daoism 1
Environmental 1
Epistemology 2
Ethics 3
Feminism 2
Hermeneutics 1
Historical ontology 1
Indian 2
Islamic 1
Japanese  2
Mathematical logic 1
Metaphysics 2
Modern 1
Moral Psychology 1
Neo-Confucianism 1
Phenomenology 1
Philosophy for Children 1
Philosophy of Language 1
Philosophy of Law 2
Philosophy of Religion 1
Political 2
Process philosophy 1
Renaissance 1

Since most UH graduates are placed into positions in non-Western fields, the blind spot is obvious—specialties in non-Western philosophy, which are almost entirely absent in the PGR evaluator pool.

I happen to be a graduate of UH and so am not entirely unbiased on this issue, but the facts are the facts, and my association with UH doesn’t change its value as a test case for assessing the inclusiveness of the PGR, using the PGR’s own standards.  Remember, the PGR does purport to represent the entire profession, not just part of it.  Surely there are other test cases as well, and concerned scholars, like David Wallace and the PGR’s own board, should look into these to see how the PGR can be improved and live up to its own aspirations for providing reliable information to all prospective philosophy graduate students.

(I know, making this claim opens me up to all kinds of sour grapes arguments: “Bruya’s test case is a laughing stock: If his alma mater doesn’t make the ranking, then the ranking must be flawed.  That’s what they all say!”  But it is a good test case—for the reasons given.)

  1. Finally, the PGR should look to the U.S. Fulbright program as a model of inclusiveness. Fulbright is one of the most prestigious fellowship programs in the U.S., but it doesn’t draw it’s awardees or its evaluators only from the most elite universities.  It draws from community colleges right up through Ivy League schools, always attempting to retain a balance.  If the PGR really aspires to represent the entire profession of philosophy by providing generalizable conclusions about what philosophers think about faculty quality in Ph.D. programs, or generalizable information about all philosophy Ph.D. programs that will place students into jobs, or generalizable information about job placement prospects into schools, then it should strive for a similar kind of inclusiveness.

Bruya’s entire reply is here.

Stella chill

guest
60 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Anonymous
Anonymous
5 years ago

This piece is brilliant. We need more like it.Report

Carolyn Dicey Jennings
Carolyn Dicey Jennings
5 years ago

In case I haven’t made this clear, your intervention in my case, linked above, made a huge difference. I am still dealing with the impact of what Brian Leiter did to me, but I know it would have been much worse if you hadn’t posted support as quickly as you did, which I think encouraged others to offer theirs. I appreciate the efforts of all those who have tried to make me feel welcome in this profession, which was a difficult charge after what I faced, but that first post meant the most to me in this regard. I have come across Brian Bruya’s work on effortless attention in my research, and so I started this current debate from a position of respect for him. Some aspects of the debate have been hard to watch, perhaps just given my history, but I am glad that Bruya has been able to focus on the intellectual dispute so far. I hope that he is far enough along in the profession to have the support he needs to deal with the rest.Report

JDRox
JDRox
5 years ago

I feel like it’s a bit rich for you to tsk-tsk Brian for impugning the motives and character of Bruya, when Bruya does exactly that to Brian, in a bit that you chose to excerpt! Recall: “When we notice that the APA specialties that the PGR would, or does, categorize under other are often associated with feminism and non-Western ethnicities and cultures, one cannot help but wonder whether the PGR’s hidden biases are based in sexism, racism, ethnocentrism, and xenophobia.” If you don’t think that Bruya was suggesting that Brian’s motives and character were beyond reproach, I’d like to be told how I’m supposed to interpret that sentence. Anyhow, I agree that impugning others’ motives isn’t a good argumentative strategy, but I would like to see strictures against it impartially applied.Report

Anon
Anon
Reply to  JDRox
5 years ago

Just want to agree with JDRox and note that Bruya basically admits what his self-interested motives are, both in the original piece and the reply. And BL didn’t just comment on Bruya’s motives, he had substantive criticisms, which Wallace has since developed in detail. It might also help make sense of the context for Justin’s comments, above, to note BL’s post yesterday about this whole matter: http://leiterreports.typepad.com/blog/2015/12/what-can-we-learn-from-the-brian-bruyametaphilosophy-fiasco.html. I don’t think I saw a link to that in Justin’s introduction to Bruya’s latest response.Report

Matt Drabek
Matt Drabek
Reply to  JDRox
5 years ago

JDRox, I haven’t read the entire dispute between Bruya and Leiter, nor do I want to. But in the bit you quote from Bruya (“…one cannot help but wonder whether the PGR’s hidden biases are based in sexism, racism, ethnocentrism, and xenophobia.”) there appears to be no questioning of motives or character, at least according to one dominant strain of thought in scholarly treatment of these matters. It looks to me like Bruya is careful to hold out sexism, racism, et al. as properties of systems rather than persons. And so the existence of a sexist, racist, et al. bias implies nothing whatsoever about the motives or the character of the person propagating the bias. Indeed, the entire point of revealing these biases is to point out to a person that they’re unintentionally advancing these systems.

Of course, in a cultural environment like the United States, where an extreme form of individualism is common, and in a field like philosophy, where the literature in moral philosophy focuses (arguably) obsessively on “moral responsibility,” it’s likely that people are going to simply misunderstand this and take it to be a personal attack. Lord knows anyone who talks about these issues regularly has been there a million times over. But your quote of Bruya read to me like a sharp contrast to what Leiter has been writing about Bruya, which is an obvious and unmistakable series of personal attacks.Report

Dude McLadpants
Dude McLadpants
5 years ago

Leaving aside the substantive issues in dispute, the answer to question #1 (Is it true that some philosophy Ph.D. programs in the U.S. use the PGR to build their programs? ) is “Yes, of course.” Anyone who says otherwise is either a fool or lying.Report

observer8
observer8
5 years ago

“the PGRs hidden motives” are obviously not Leiter’s motives given a) Leiter is not the PGR, and b) the PGR is an entity composed of numerous agents and entities, not a conscious being with motivations. A more apt way to express the thought would be that the PGR embeds endemic racism, sexism, and other cultural biases, which it surely does.Report

JDRox
JDRox
Reply to  observer8
5 years ago

Leiter is obviously not the PGR, but Bruya’s article pretty clearly targets Leiter, and as Bruya pointedly notes (in response to an argument that criticism shouldn’t be directed at Leiter personally), “Brian Leiter… handpicked his original slate of evaluators”. He goes on to say that “Leiter seems to take exception to the fact that I target most of my arguments at him rather than at the PGR as a publication.” So it’s fairly plausible that the “PGR’s hidden motives” are supposed to be Leiter’s hidden motives. And that was my only point. However, even if they’re not, they’re still someone’s hidden motives, right?

Well, that brings us to your second point. You say ‘the PGR embeds endemic racism and sexism’. I’m not sure how to interpret that claim, but offhand it seems to require the existence of racists and sexists to be true. If you’re denying that, I’d be interested to know what you take the truth-conditions (truth maker, whatever) to be for ‘the PGR embeds endemic racism and sexism’. I’d also be interested in hearing whether you think that sentence has the same truth-conditions (truth maker, whatever) as “the PGR’s hidden biases are based in sexism, racism, ethnocentrism, and xenophobia”, which, again, seems to me to require the existence of people–people influencing the PGR–with bad motives or character to be true. ()Report

Nick
Nick
Reply to  JDRox
5 years ago

““the PGR’s hidden biases are based in sexism, racism, ethnocentrism, and xenophobia”, which, again, seems to me to require the existence of people–people influencing the PGR–with bad motives or character to be true.”

Also, in your replies you seem to equate “the PGR’s hidden biases” with “the PGR’s hidden motives”.

Bruya’s statement does not require the existence of people with bad motives or character to be true. People with good motives and good character may be influenced by implicit biases that are racist, sexist, ethnocentrist, and/or xenophobic. I took Bruya to mean something like this, but I can see how one could have your reading too.Report

observer8
observer8
5 years ago

I didn’t read Bruya’s piece as “targeting” anything except the PGR, which in fact he has done consistently even as Leiter has chosen to “target” Bruya as a person with bad motives, bad faith, committing fraud, etc. And racism, sexism, etc. Are biases possessed by humans, obviously, which even when unknown to the agents possessing them are often expressed unconsciously through institutions like the PGR. So no, none of those comprising the PGR need have a bad motive for the PGR to embed racism, etc.. I read Bruya as suggesting that is precisely the case, and not as saying that Leiter is racist, which requires conflating Leiter with the PGR (which you do if you see him “targeting” Leiter rather than the PGR, obviously), and confusing racism etc. with states of affairs requiring conscious intentions.Report

David Wallace
David Wallace
5 years ago

For the most part, I’m happy for readers to make their own judgements as to whether Prof. Bruya adequately responds to my criticisms. My arguments, and Prof. Bruya’s response to them, are probably not clarified by further back-and-forth.

I will, however, make these points:

1) It is deeply disturbing that Prof. Bruya doubles down on his relegation of his methodology to an appendix, and his appeal to a footnote as a justification. You just *cannot do this* in serious, quantitatve scholarship. You have to make it explicit, up-front, crystal clear, what your methodology is – *especially* when you are using categories acquired from your critical target and rearranging them. Prof. Bruya says that this is “perfectly reasonable to me”. It is not “perfectly reasonable” by the standards of any quantitative-science journal I have ever read. (I welcome counter-examples.) In the absence of such, I continue to think that Bruya’s coyness about this methodology (and his coyness about his methodology on evaluators’ institutions, which he has not yet addressed) is inappropriate as a matter of scholarship.

2) Prof. Bruya resorts to the very worrying methodology of saying that my criticisms are only valid insofar as they also apply to the PGR. (He goes as far as to italicise the points where he feels a double standard is applied.) *This is irrelevant.* Either the reasoning in his paper is valid, or it isn’t. You can’t defend shoddy science by pointing out that others’ science is also shoddy. It’s fine to impugn my motives and object that I only spend time on criticisms of the PGR. (The truth is that the PGR’s defenders don’t make clearly-false mathematical claims, but we can dispute that.) But the question of what I bother to spend my time criticising has nothing whatever to do with whether a given criticism is valid. Does Prof. Bruya think his arguments are valid, or not? If so, let him defend them. If not, it’s irrelevant whether other arguments are also invalid.

3) Prof. Bruya notes that I have not addressed all of the objections he raises. His criticism predates the extension of my criticism to cover his conflation of two categories of institutional association (addressed only in a graph x-axis and never in the text). I didn’t engage with the main additional point he raises (regression analysis of PGR-vs-all-subject-aggregate mark) because I lack the data. If Prof. Bruya is prepared to email me his data-set ([email protected]) I’d be very happy to play with it, with the proviso that I have a baby turning up in the next few days!

4) Prof. Bruya is legitimately concerned that I didn’t give a proper data link for my PDC data. Mea culpa. Here’s a daily nous link: http://dailynous.com/2015/10/07/philosophers-by-subject-area/

5) Justin Weinberg has chosen to repeatedly give Prof. Bruya space to defend his views in the main posts on DN, leaving critics to make their points in the comments and/or in linked documents. That’s entirely his prerogative but is perhaps regrettable.

Let me finish by saying that I’d rather hoped this could be sorted out in a collegial fashion. Prof. Bruya’s comment suggests otherwise; on that basis, I’ve communicated with the editor of Metaphilosophy and called for the paper’s withdrawal. My email to that effect is at https://dl.dropboxusercontent.com/u/8561203/bruya%20letter%20to%20editor.docx .Report

Tim Kenyon
Tim Kenyon
Reply to  David Wallace
5 years ago

“It is deeply disturbing that Prof. Bruya doubles down on his relegation of his methodology to an appendix, and his appeal to a footnote as a justification. You just *cannot do this* in serious, quantitatve scholarship.”

Perhaps I’m misunderstanding the shape of the complaint, but in my experience it is not uncommon to have much of the methodology of an empirical paper relegated to an appendix, with a footnote observing this. Here, for example, is a recent 6-page piece of quantitative scholarship, and its 30-page footnoted appendix on methodology:

http://www.pnas.org/content/112/17/5360.full.pdf
http://www.pnas.org/content/suppl/2015/04/08/1418878112.DCSupplemental/pnas.1418878112.sapp.pdf

This example comes to mind because when I first read it I missed a key methodological detail given only in its appendix, and was for a time convinced that the authors had overlooked an important confound that would have really weakened their interpretation of the results. That can happen! It’s fair to say that, 30-page appendix or not, there is a lot of methodology reported in the PNAS 6-pager anyhow. There’s a lot in the main body of Prof Bruya’s paper as well, but maybe you think there’s just not enough, or anyhow not the key points, contained there. Clearly, though, this is a matter of degree even within empirical journal publication, and not some sharp violation of a commandment of quantitative scholarship never to relegate methodology to an appendix, or to only place inessential details there.

What should one make of this perception of degree, then? Not a publicly posted request that the editor withdraw the paper, I think. This seems a good time to remember that we’re discussing, not *whether*, but *where*, a paper in a philosophy journal explains its methodology. In my view, you were more clearly on the right track early on in your comments on this matter, when on a different thread you allowed that philosophy publication norms and conventions are not the same as scientific journal norms and conventions. That remark struck me as reflecting an admirable sense of perspective and collegiality in disagreement.Report

David Wallace
David Wallace
Reply to  Tim Kenyon
5 years ago

Of course you can put details of a long complex methodology into an appendix. That’s very different from failing to clearly outline what you are doing.

On norms and conventions: I don’t think it’s *okay* for a philosophy journal to have different norms for quantitative data than a science journal; what I meant and mean is that I don’t think it’s *blameworthy* for those norms to be violated.Report

WP
WP
Reply to  Tim Kenyon
5 years ago

It goes beyond relegating the methodology to the appendix and not flagging a likely misunderstanding in the text—the paper actively portrays the areas as the ones used by the PGR’s evaluators. The ‘Area Dilution’ section begins with “Consider the areas listed by the evaluators as their own areas of research…” and then continues to refer to them as the areas evaluators listed throughout the section. The table’s headings are “No. of Times Area Listed in Total,” “No. of Times Area Listed With M&E.” But the column does not show that. It shows the number of times an area was listed with M&E OR philosophy of science OR logic.

As labeled, the information in the table just isn’t true. M&E was listed 157 times, not 185. Report

David Wallace
David Wallace
Reply to  David Wallace
5 years ago

On reflection I think I spoke too soon when I said that I couldn’t engage with Prof. Bruya’s regression analysis without the data. It also appears to have a severe methodological flaw, though I don’t think this is as serious a point as others I’ve raised.

Prof. Bruya calculates the difference, for each US institution, between (i) the PGR rank and (ii) the “Bruya rank”, i.e. the equally-weighted sum of all speciality marks. He then runs a linear regression analysis to determine the dependence of this difference on the equally-weighted sums of speciality ranking in four areas: History, Value, “M&E” (being Prof. Bruya’s combination of the PGR categories of M&E and Science) and Other.

The problem is that these are very strongly correlated variables: generally speaking, departments with high PGR ranks in one category have high PGR ranks in others, and running regression analyses on very strongly correlated variables has to be done with great care.

To illustrate: suppose I’m interested in whether certain things about US voters (let’s say, their salary) can be predicted by their voting patterns. If one of my regression variables is “usually votes Republican in Senate elections” and another is “usually votes Republican in House elections”, then it’s very difficult to determine the relative contributions of those two variables to a voter’s salary, since each of them is a very good predictor of the other.

You can work around this with a sufficiently large dataset, and there are statistical tests that can be done to ascertain whether a result is still significant even given a high correlation, but Prof. Bruya does not mention any such test.

Methodologically, what Prof. Bruya should have done is run a regression analysis using the *difference* between average speciality ranking in a given area and average speciality rankings overall, since those variables are much less strongly correlated. Had he done so I suspect he’d actually still have found the effect he discusses, since looking at Kieran Healey’s analysis of the PGR certainly suggests better returns to specialisation in M&E (though less obviously in Science) and relatively poor returns to specialisation in History. But I can’t know this (not without getting Prof. Bruya’s dataset, or doing a long data-entry exercise of my own) and it’s certainly unsafe to conclude it from Prof. Bruya’s analysis.

(If anyone’s wondering how I know that Prof. Bruya didn’t actually do what I’m suggesting he should have done, the answer is (i) that’s not what he said he did; (ii) if he did, he’d have to have left out one of the larger categories, since they’d sum very close to zero for any institution, and run the analysis on the others. Otherwise the correlation between the variables would be even tighter – close to linear dependence. But he reports R-squared values for all the categories.)Report

David Wallace
David Wallace
Reply to  David Wallace
5 years ago

Added this to my main document, for anyone still keeping track. Main document still at https://dl.dropboxusercontent.com/u/8561203/bruya%20critique.pdf. v2 at https://dl.dropboxusercontent.com/u/8561203/bruya%20critique%20v2.pdf Report

Grad student
Grad student
5 years ago

I really have a difficult time appreciating that anything other than persons can be racist, xenophobic etc. You can’t just take a bunch of non-racists, give them a job, and end up with the result being racist. *Somebody* in the mix is racist, or at least prejudiced, and I can’t read Bruya’s comments any other way, not on standard interpretations of “racist.”

Even “Endemic racism and sexism” that the PGR embodies still strikes me as something that requires there to be racists and sexists, and I have no idea what Bruya could possibly mean by his comment if it isn’t that some people in the assessment pool are racist, sexist, etc. Again, how does an institution end up racist or sexist when it does not consist of any racist or sexist individuals? Mysterious “forces” of racism and sexism nudging people? Perhaps, but still I think you’d have to call those people susceptible to racist and sexist influences racist and sexist—no?

All that said, I feel like the discipline as a whole is to blame for its assessors being racist or sexist, not the assessors themselves or the assessing program. The country club analogy above is wrong. The PGR critique is certainly not talking about changing the way one country club operates. It’s like writing a letter to the editor of “Country Club Monthly” complaining that they rank country clubs based on evidence that doesn’t reflect the interests of marginalised groups, to which the appropriate response is “ok then go make your own or go and change the way those clubs operate, and we’ll rank accordingly when that happens,”Report

Matt Drabek
Matt Drabek
Reply to  Grad student
5 years ago

Hi, Grad Student. I suppose it’s all well and good that you don’t understand how things other than persons can be racist, et al. But whether or not you understand the claim isn’t relevant. What’s relevant is that the claim is a common one (e.g., it’s probably the predominant view within critical race theory, for instance), and it looks to be the most plausible reading of Bruya. At least those lines I quoted above.Report

Patrick Mayer
Patrick Mayer
Reply to  Grad student
5 years ago

“I really have a difficult time appreciating that anything other than persons can be racist, xenophobic etc. You can’t just take a bunch of non-racists, give them a job, and end up with the result being racist. *Somebody* in the mix is racist, or at least prejudiced, and I can’t read Bruya’s comments any other way, not on standard interpretations of “racist.””

OK, so you are familiar with Adam Smith’s invisible hand right? Where a bunch of people with no intention to help the poor but only with intentions to profit engage in actions that lead reliably to the lowering of prices on the goods that the poor need. So you have some attitude which no members of a group have, but where the group , because of the way the relationships between the members of the group are structured (competitive relationships marked by non-domination of the market by any buyers or sellers, symmetrical information, etc.), engages in activities which are what you would expect of them if they did all have that attitude. It might be the case that people working in philosophy of race mean more by ‘institutional racism’ than this, but we can all appreciate at least that what goes for altruism in Smith’s invisible hand example can go for racist attitudes to. Even if the invisible hand situation doesn’t actually occur much because the conditions of perfect competition aren’t regularly met, you understand how they could happen.
And you can come up with cases where the way the relationships between the members of a group are structured lead to the exact behavior you would expect if the group was full of racists, even though none of them actually are. So we fund schools in the United States via property taxes, which because of the background housing segregation and the long term wealth effects of slavery and Jim Crow, leads to underfunding of schools attended mostly by people of color. Did we set up the school funding system with the intention of doing that? Probably not, that would take more foresight than most anyone has. But will this reliably lead to poor treatment of people of color? Yes. Many cities evaluate the leadership of their police departments by raw numbers of arrests made. Because Americans can sue the police, and because people with money have a greater probability of suing police if they are arrested, this means that our method of evaluating police departments gives them an incentive to make arrests in poor areas, which for reasons similar to those in the case of school funding, means that we have police officers focusing on arresting people of color more. This is what you would expect from a bunch of racists, but importantly they don’t need to be racists. A black officer can make arrests in a black neighborhood because his captain told everyone they needed to get their numbers up. That captain could have done that not out of racist attitudes but out of a desire to keep his job because the commissioner told him he had to. And the commissioner might have done it because of the mayor, who gave the command because he had to please suburbanites with no idea what is going on in the city but who elect most of the state representatives the mayor has to beg for money. And at this point just watching the Wire would be so much better for you than reading what I am writing that I am going to wrap it up.
Now a barrier to appreciating this would be an insistence that ‘racism’ is the name of a moral vice, that it is a term used to make the moral failing of an individual. Because of course in these cases it might be that no particular person has that moral vice, so none of their actions can express that moral vice, whether these actions come in the context of a group or not. But this is just the wrong concept of racism to have. It is of much less practical value than one which focuses on patterns of behavior (so that you can predict harms that will be done) rather than character flaws.Report

Grad Student
Grad Student
Reply to  Patrick Mayer
5 years ago

I’d like to thank you for the honest and thoughtful answer. So while on my shall we say “moralized” conception of racism—the only one I had been familiar with until very recently—I suppose my complaint could still hold, but at least on a non-trivial amount of interpretations of Bruya’s comment, there’s no reason to interpret the comments so narrowly as I did. Seeing it as an invisible hand type-systematic-endemic-institutionalized-unintentional-racist/sexist/xenophobic etc-tendencies-and-behaviors kind of comment is far more charitable and probably more in-line with ideas and areas of research with which I’m unfamiliar.

I still wonder about two things though, 1) the truth of the claim, and 2) if it is true, then what is it indicative of? I don’t know the answer to 1), but if the charge sticks, then I’d say, to answer 2) that the problem goes much deeper than the PGR. Your examples of institutionalized systems of racism without racist agents per se is compelling, but I think the apt analogy would have to be with the discipline as a whole, and not necessarily reports on it (PGR). It’s not like philosophy as a discipline is all wonderful and the big bad PGR has its own behavioral tendencies that misrepresent the field as a whole because of (perhaps invisible) assessor tendencies towards racism/xenophobia and the like. The PGR only exemplifies these tendencies because they’re alive and well in philosophy, and that’s the bigger problem. This is why I think Bruya’s other article in *Dao* (the article the one in Metaphilosophy was supposed to be an appendix to) is far superior to this one. At worst, this article is all the things Wallace and Leiter say it is. At best, it’s misplaced, since the field as a whole needs to be fixed in these respects, not the messenger.Report

Patrick Mayer
Patrick Mayer
Reply to  Grad Student
5 years ago

If I was of help then I am glad. I won’t take a position on how this applies to the PGR, except to say 1) you are right that if there is institutional racism it is going to have to involve more people than the advisory board but that 2) that it needn’t include all of philosophy, at least not to the same degree. There are support systems (mostly but not entirely informal from what I can tell) in place for under-represented groups in philosophy. The people who constitute these systems are more likely to be aware of institutional factors that lead to the poor treatment of philosophers from under-represented groups, as they are the people on the receiving end, and are more likely to not be taking part in them and more likely to be actively opposing them. So while it can’t be just the PGR board it needn’t be everyone equally. The accusation here, I take it, is that the PGR board represents the philosophers who are not paying sufficient attention to the problems and so are more likely to be reproducing the problem (in the limited ways the PGR can, by guiding judgments of prestige) without having any intention of doing so. That is not my accusation. I haven’t thought hard about whether it is true.Report

the Onion Man
the Onion Man
5 years ago

I have complained frequently in this space about the provincialism of the academic Philosophy world, which relegates the work I do with Sanskrit and Tibetan Buddhist texts to second- or third-class status. So in some ways I am quite sympathetic to the critiques of the PGR.

At the same time, I can’t help but agree with David Wallace and JDRox, to the effect that apparently name-calling is just fine as long as you are calling someone the right kinds of names (and no, chalking up “racism and sexism and xenophobia” to systems rather than certain individuals is not meaningfully different from calling those individuals racist, sexist, and xenophobic). And I find Justin’s editorial decision, as far as I can tell, to allow only one side of this discussion to occupy prime real estate on the front page of DN extremely problematic. In fact, I find it exactly the same kind of problematic as the alleged issues with PGR.

Tangential question: having read hardly any of this exchange, could someone explain to me why, “if you understand the working of the Gourmet Report, the answer is clear: you should hire the man”? This makes no sense to me and seems to rely on a long chain of assumptions that have no relation to the PGR itself. But I would like to understand this argument.Report

Anonadjunct
Anonadjunct
5 years ago

Justin,

I’m not sure why you won’t admit to having a dog in this fight. You’ve guest posted two of Bruya’s lengthy replies after posting Lebon’s criticism of Leiter. At least Bruya’s post followed up on an article. The Lebon piece came about, I gather, because Leiter disagreed with Jason Stanley (who has somehow become the voice of marginal philosophers? Whatever, the other half really does live differently, I guess ) on Facebook. Leiter is a notable philosopher, but not so notable that it is important news for the discipline whenever someone gets pissy with him. I don’t think anyone believes that you just happen to find these posts newsworthy.

Further, you literally started putting up posts and guest posts against the idea that there were too many PhDs (and too many adjuncts) immediately after Leiter starts pushing that idea. There is nothing wrong with being in opposition to Leiter and his blog ( that’s partly why I come here), but be honest about it.Report

Jiminey
Jiminey
5 years ago

Of course the “collegial” manner of dealing with disagreements about methods where there is no evidence of fraud (as indeed, despite the bandying around of the word there does not appear to be unless you can demonstrate something more than you’ve shown) would be a letter to the editor of the journal, another paper critiquing the paper, or some formal request for inquiry regarding a breach of scientific integrity rather than public accusations and slurs on blogsReport

David Wallace
David Wallace
Reply to  Jiminey
5 years ago

@Jiminey: I haven’t accused anyone of fraud. It’s defensible to suppose that I should have raised all this privately. But all came out of a public discussion on DN, and I originally began commenting on the methodology after a guest post from Prof. Bruya himself, so I think it’s also defensible to regard this as a legitimately public matter.

@another anon: in fairness to Justin, I’ve been commenting fairly prolifically here and nothing I’ve written has been withheld.Report

another anon
another anon
5 years ago

I just want to second the calls for Justin to be an honest broker here. I accept that it’s his blog and he can do as he wants, but no one should labor under the delusion that the comments section here represents the full range of opinions on these and other matters . I’ve had quite a few fairly minor comments witheld, whereas Justin lets through any number of posts accusing Leiter of being racist, xenophobic, etc etc. No wonder folks take to the metablogs.Report

Pendaran Roberts
Pendaran Roberts
5 years ago

David Wallace should be given space to present his position in a main page post on DN.
I am assuming what we all care about is getting at the truth. If so, both opposing parties should be given space to debate their positions.

Report

David Wallace
David Wallace
Reply to  Justin Weinberg
5 years ago

Considering that the bulk of discussion at both this and the last Bruya thread has been dominated by people discussing my objections, I don’t think there’s any question that I’ve had plenty of space to reply! I’m not particularly seeking a guest post; on reflection I probably take back my point (5) in my original comment on this thread.

(As a minor point, I don’t think I have a single “main criticism”. Most of the discussion has been about one particular issue (categorisation) but that was just one of several concerns.)Report

Diyas
Diyas
5 years ago

I belong to an under-represented group in philosophy. For context, I will divulge that I am seeking admissions into graduate programs this season. My comment may not have any practical effect on the profession, but I think its utility will consist in illustrating the perspective of a minority student who agrees that the Philosophical Gourmet Report embodies racism and sexism. First, I will make some claims: my minority status is not an argument. I aim to justify one claims on reasonable grounds, that will be difficult to make explicit because not all the facts about evaluators are available.

(Claim 1): The PGR embodies racism and sexism. What sense of embodiment are we talking about here? Racism and sexism are prejudices based on fallacious reasons. Embodiment is the idea that something can express or represent an idea, and that the expression or representation of that idea has some real consequence. If something embodies racism and sexism, then that thing expresses or represents prejudices in such a way as to consequentially affect reality. Does the embodiment necessarily ontologically depend upon racist or sexist people? No, but it does necessarily ontologically depend upon the existence of actions or events that were given rise to by racist or sexist motivations; the motivations could be hidden or ignored by the people that carry the actions out, but the ignorance of the racist or sexist motivations does not mean that the people are themselves racist or sexist.
There are degrees to which racism and sexism affect the world. We can separate blatant prejudices from insidious prejudices. I think that the PGR embodies insidious prejudices that originate from stochastic facts about people who rank departments. The stochastic facts include: facts about evaluators’ racial, gender, and class privilege, facts about who evaluators like and who they do not like, facts about which department impressed them and which ones did not based on heuristic reasoning, and facts about the reason why evaluators got into the authoritative position in the first place–was is just their sheer brilliance, networking skills, etc that led them to their success and position of authority? Maybe, but not always–sometimes, their authoritative positions are grounded partially in positive discrimination — there are more male evaluators than female evaluators, more white evaluators than non-white evaluators, more evaluators who could afford to attend expensive or prestigious universities the pedigree of which helped them achieve success in their studies.
For all this, I submit that the racism and sexism embodied by the PGR should be understood in this sense: there are permissible generalizations that those who evaluate departments share certain facts about their life that are grounded in positive discrimination over their skin color, gender, class origins, and whatever other arbitrary secondary characteristics that help people succeed in this world.

Whether what I submit is reasonable or not, I am unsure. It is just my opinion, and I would appreciate any objection against it when it comes to how we think about the PGR as embodying racism and sexism in the sense that members of over-represented groups owe some of their success to positive discrimination.

Report

Jiminey
Jiminey
5 years ago

David Wallace: kudos for not saying “fraud” and addressing your letter to the editorial board which I hope they publish as a letter to the editor. Leiter has been less responsible, accusing Bruya of fraud repeatedly. That’s a shame and pollutes the proper process for this sort of dispute. Absent fraud, the scientific record should be clarified by a trail of correcting publications. Retracting the paper would be unwarranted. Perhaps Bruya will run a regression analysis as an addendum. Hopefully you will submit a paper of your own. There are any number of ways to establish a record of scholarship in this matter that will be useful for future scholars of the subject, none of which require threats of lawsuit, cleansing the record of a (even an erroneous, but so far just disputed) paper. This is an important topic for philosophy and it should be dealt with thoroughly and not have the dialogue and possibility of an expanding body of scholarship chilled by loosely flung invective, threats, and oersonal attacks. I’m glad that Wallace and Bruya are documenting their positions in a scholarly manner and am interested in how scholarship proceeds in this area in the pages of appropriate journals. Hopefully as a result we’ll improve our ranking methodologies if we decide they are worthwhile at all.
Report

Jamie Dreier
Jamie Dreier
5 years ago

Jiminey,

“Absent fraud, the scientific record should be clarified by a trail of correcting publications. Retracting the paper would be unwarranted.”

Can you explain why? Scientific journals retract papers quite often for honest mistakes; here is an example. Maybe this practice is wrong and all of those retractions are unwarranted, but it really is the standard practice.Report

JDRox
JDRox
5 years ago

@ Matt Drabek: Thanks for the response: I shouldn’t have used strong language like ‘require’. I am happy to grant at least for the sake of argument that there *can* be, e.g., racism without racists. So I agree that Bruya’s claim doesn’t strictly imply that Brian or anyone else had bad motives or character. But I do think it strongly suggests it, roughly because the existence of racism in some domain strongly suggests the existence of racists. (This depends the fact that racism is wrong. If we use ‘racism’ in a way that doesn’t entail that it’s wrong, then all bets are off.) But see below for a bit more.

@ Nick: You say that “Also, in your replies you seem to equate “the PGR’s hidden biases” with “the PGR’s hidden motives”.” The person I was responding to equated those, I was just responding to that comment. I agree that these are different. You also say that “People with good motives and good character may be influenced by implicit biases that are racist, sexist, ethnocentrist, and/or xenophobic.” I agree about the motives part, but I’m not so sure about the character part, unless you’re just saying that we’re all flawed, no one is perfect, so one can be a “good” person even if one is bad in various ways. But if I have implicit biases that are racist, sexist, etc., then *something* is wrong, or less than ideal, or whatever with me. Right? And it seems pretty natural to say that the thing that’s wrong, or less than ideal, is something to do with my character. If orthodoxy says otherwise, I’d be interested in hearing the story.

@ Justin: You say, “I understand that the “feud” frame is entertaining, but it is also silly and distracting. The discussions here would be better if that frame were left aside.” Well, I thought it was obvious that there was a feud. You seem to go out of your way to act as if Leiter’s blog doesn’t exist: you don’t ever link to it (Right? Even when there’s obviously good content there.), and whenever you write about Leiter you sarcastically introduce him as if no one has any idea who he is. I’m not saying that your behavior is objectionable in any serious sense, but that’s why I, at least, would think that “feud” framing was appropriate.

@ Observer8: First, Bruya basically admits that he targeted Leiter, as I tried to show above. Second, while I do admit that there *can* be bias without biased people, I think it’s unlikely, and I think you should think it’s unlikely, that there’s bias without biased people (And I think biased people have flawed characters–I think that various biases are flaws in character.) After all, in the vast majority of cases where racism exists, racists are playing a big part in it. Right? (Again subject to the proviso mentioned above that racism is something bad.)
Report

Nick
Nick
Reply to  JDRox
5 years ago

1. “But if I have implicit biases that are racist, sexist, etc., then *something* is wrong, or less than ideal, or whatever with me”
2. “And it seems pretty natural to say that the thing that’s wrong, or less than ideal, is something to do with my character”

I agree with 1, in the sense that one should not *want* to hold implicit biases. But they don’t just go away by wishing them away; thats the whole point.
The extent to which 2 (or something similar to 2) is true is debated in the literature on implicit bias, as I understand it. Its not really my area, but you can look it up if you are interested. See this for example: http://philpapers.org/rec/HOLRFI
Report

another anon
another anon
5 years ago

@Justin – Actually you’ve censored multiple comments of mine, one of which is now posted as an addendum to a post on Leiter’s blog (http://leiterreports.typepad.com/blog/2015/12/what-can-we-learn-from-the-brian-bruyametaphilosophy-fiasco.html#more).

If that mild comment offends your prim sensibilities, then fine, it’s your blog. But you regularly let through all sorts of comments (let alone full posts) stating or obviously implying that Leiter is racist and sexist and just a big bad bully, often without serious evidence. (And your assertion that you also censor some comments even more critical of Leiter isn’t really relevant here, since enough such comments already make their way through). Thus your claim to be a neutral arbiter rings a bit hollow.

Whether or not there is some sort of personal “feud” between you and Leiter, there are clearly plenty of philosophers online who, for various but often related reasons, wish him ill and denounce others who do not, and the DN comment thread and main page cater to these folks. It’s just helpful to be up front about this, lest someone think that the comment thread here is indicative of more than just a small but vocal segment of the profession.Report

Jiminey
Jiminey
5 years ago

Hi Jamie,

I am linking to a useful guideline for journal editors to issue retractions, and usually authors themselves do the retraction not journal editors. In the case of the paper at issue in all our discussions here, the claims of “errror” have not yet been agreed to, as the various parties make claims to proper though differing methodologies. There is no evidence of misconduct, and if there is a way to correct any error that might be domonstrated that is preferable. In any case, retracted articles must remain available for perusal even after retraction: http://publicationethics.org/files/retraction%20guidelines.pdf

a relevant quote from those guidelines “Retraction is a mechanism for correcting the literature and alerting readers to publications that contain such seriously flawed or erroneous data that their findings and conclusions cannot be relied upon. Unreliable data may result from honest error or from research misconduct.”

So far, we have experts disagreeing, not seriously flawed or unreliable data. Perhaps subsequent studies can clarify whether there is clear and unreliable data, or whether there are honest disagreements as to approrpaite methodologies. Report

JDRox
JDRox
5 years ago

Justin: I can only say that I thought, based on your linking/posting patterns, that you positively tried not to link to Leiter’s blog. Maybe I should say, unless absolutely necessary. That’s your prerogative, of course. And I thought the introduction thing was humorous! Anyway, I honestly though that you and Brian were explicitly feuding, and that your blog and his were engaged in a sort of feud. But of course there are many things I honestly believe that are just not the case. If I’m misinterpreted you, I apologize. The claim that you were feuding wasn’t intended to be any sort of insult or critique.Report

Art Grymes
Art Grymes
5 years ago

Justin wrote: “For what it is worth, I twice this past week invited the current editor of the PGR to join the discussion here. However, I just learned this morning that for unrelated and completely justifiable reasons she has been largely offline lately.”

On a charitable reading, Justin is not implying that the current editor of the PGR owes us justification for being offline. But I’m genuinely curious: do people think it is her (obviously defeasible) duty to explicitly weight in? Or does she have the (weaker) duty to read the criticisms and do something about those that she considers well-founded before publishing the next PGR? It’s not clear to me that she has either duty, though the latter certainly seems like a good idea.Report

Matt Drabek
Matt Drabek
5 years ago

Hi JDRox,

Thanks for the reply. I imagine Bruya’s input would be just about the only thing that would settle whether his suggestion about the PGR’s racism implies actual racist persons working for the PGR. There’s a broad range of definitions of racism – some that define it in terms of an individual person’s prejudices or discriminatory beliefs and/or actions, some that define it solely in terms of social systems, some that fall in between, or some that allow all of these things. If Bruya comes down more on the social/systemic side of things, then there wouldn’t be any suggestion that Leiter is a “racist,” because when you go far enough down that part of a continuum of definitions there’s no such thing as a racist person. Calling a person a racist would be applying the term to the wrong sort of thing.Report

Minh Nguyen
5 years ago

My apologies in advance if the points have been stated before. I’ve been in the business since 1991 and I have to confess, with dismay, that this is the first time that I’ve seen a serious philosopher (whom I respect for his consistently thoughtful contributions to various philosophy blog discussions) call on the board of a philosophy journal to withdraw a paper. I wonder if we’ve got all worked up over Prof. Bruya’s paper because it concerns a contentious and sexy issue such as the PGR. Whatever methodological flaws if any of Prof. Bruya’s paper, to ask that it be retracted seems exceedingly harsh to me, especially in the conceded absence of “malice,” “deceit,” “fraud,” “gross negligence,” “bad faith,” or the like. In my view, one has to be very careful about this non-“collegial” and “confrontational” approach (all quoted terms are Prof. Wallace’s). Given the profound and widespread disagreement among philosophers on virtually everything philosophical, our relatively high fallibility regarding such matters, and the increasing reliance on the part of many of us on “data,” I strongly urge charity, humility, caution, and of course rigor and more rigor. Relatedly, if we are to retract papers for honest mistakes (as some seem comfortable with the idea), then in my opinion, (virtually) all philosophers whose work I’ve seriously studied ought to have at least some aspects of their work retracted.Report

David Wallace
David Wallace
Reply to  Minh Nguyen
5 years ago

@Minh Nguyen: This is a reasonable question. And of course retraction has to be a higher bar than honest error.

Here’s where I am coming from. Methodologically this is not a philosophy paper: it’s a quantitative social-science paper, dominated by statistical analysis, claims that various results are demonstrated by data, discussion of correlation coefficients, presentation of regression analysis, and the like. (It describes itself as a “data-driven critique”.)

And all of this is fatally flawed by the standards of the quantitative sciences. A central methodological assumption is relegated to appendices and described in a way that naturally invites confusion. Another is not stated at all, to be inferred only indirectly, and the quantitative analysis and the discussion rely on incompatible versions of that assumption. A mathematical tool is used in a context where it is normally inapplicable and no discussion is given of its validity. Classifications of obviously uneven sizes are treated as demographics. And so on. Nothing of this kind would ever have got past peer review in a serious science journal; indeed, science papers get withdrawn for much subtler methodological mistakes.

As I have repeatedly noted, I don’t hold Prof. Bruya blameworthy for this. I had hoped that when I pointed it out in some detail he would have responded by withdrawing the article or at any rate publishing a correction that acknowledges the flaws in most of the “data-driven” part of the paper. (That’s entirely compatible with continuing to criticise the PGR on the other grounds that Prof. Bruya raises.) Instead he has doubled down on his analysis. In this situation I didn’t, and still don’t, see any way forward less drastic than a call for withdrawal, if this issue is to be clearly identified as a simple failure of scientific methodology, and not just treated as one more philosophical dispute where reasonable people can differ, and/or annexed to the interminable disputes about the PGR.

You say: “Given the profound and widespread disagreement among philosophers on virtually everything philosophical, our relatively high fallibility regarding such matters, and the increasing reliance on the part of many of us on “data,” I strongly urge charity, humility, caution, and of course rigor and more rigor.”

The point is that these are not *philosophical* matters, and they ought not to be controversial. The only reason to have a high fallibility regarding them is lack of understanding of statistics and scientific methodology. As for a reliance on data, *mostly* when philosophy engages with data it does so at one remove, through philosophical engagement with statistical analysis in the science literature. (I have no objection to people quoting R-squared values without much methodological discussion, if they’ve got those R-squared results from a published paper). But if we as philosophers are going to roll our own analysis, we absolutely have to hold ourselves to the standards of science here.

I have taken a large amount of time and energy to engage with this issue at an inconvenient time for me not because I particularly want to get involved in a debate about the pros and cons of the PGR but because it is important to me that philosophy, as a discipline, is able to engage with science properly and in a way that can reasonably ask to be taken seriously. If we’re prepared to let this kind of thing appear uncorrected in our research literature, and to suppose that straightforward and widespread error on elementary matters of methodology is just one more form of philosophical controversy, we will make fools of our discipline. We will appear to others as if we are children playing with tools that we do not understand.
Report

Minh Nguyen
Reply to  David Wallace
5 years ago

Dear Prof. Wallace,
Thank you very much for having taken some time and energy to respond to my post at an inconvenient time for you.
Let me reiterate my admiration of and respect for your consistently thoughtful contributions to various philosophy blog discussions.
With regard to your call for the board of METAPHILOSOPHY to withdraw Prof. Bruya’s paper, I’m afraid we have a fundamental disagreement.
I’d like to be clear at the outset that I make no claims regarding the merits or demerits of the PGR, Prof. Bruya’s paper, or your critique of his paper.
You wrote: “Am I calling for a retraction? That sounds too confrontational. I’m pointing out some severe methodological problems in a published paper in the hope that the reasonable and professional people involved with the paper’s publication (not least its author) will act sensibly in light of those problems.”
And: “I’d rather hoped this could be sorted out in a collegial fashion.”
Somewhere along the line, the hope has become something like a threat of the form: “If Prof. Bruya doesn’t do X [“withdrawing the article or at any rate publishing a correction that acknowledges the flaws in most of the ‘data-driven’ part of the paper”], I’m going to do Y to him [“calling on the board of METAPHILOSOPHY … to withdraw this paper”].”
If this is your idea of being “collegial” or non-“confrontational,” I’m afraid I don’t share that idea.
Has it ever occurred to you that your critique of Prof. Bruya’s paper may be mistaken?
That what you consider to be “uncontroversial,” “straightforward,” “elementary,” etc. may not be so to others?
That it may take some time for one to recognize one’s own errors?
Why not give yourself and Prof. Bruya more time to think more about the matters in question?
Why not submit your critique of Prof. Bruya’s paper for consideration for publication in METAPHILOSOPHY?
I’m sure the Editorial Board is very much aware of this affair and will assign your submission to the most competent referees so as to do justice to both you and Prof. Bruya.
Why, in short, call for a retraction when you yourself originally considered it “too confrontational”?
There is too much tension and confrontation in our profession already.
I hope you, Prof. Bruya, and others will find a way to satisfactorily resolve all this in a professional and peaceful manner.
Lastly, and most importantly, you wrote that you “have a baby turning up in the next few days!”
Very best wishes and congratulations in advance on becoming a father (again).
Cheers!
MinhReport

WP
WP
Reply to  David Wallace
5 years ago

“A central methodological assumption is relegated to appendices and described in a way that naturally invites confusion. Another is not stated at all, to be inferred only indirectly, and the quantitative analysis and the discussion rely on incompatible versions of that assumption. A mathematical tool is used in a context where it is normally inapplicable and no discussion is given of its validity. Classifications of obviously uneven sizes are treated as demographics. And so on. …

As I have repeatedly noted, I don’t hold Prof. Bruya blameworthy for this. I had hoped that when I pointed it out in some detail he would have responded by withdrawing the article or at any rate publishing a correction that acknowledges the flaws in most of the “data-driven” part of the paper. … Instead he has doubled down on his analysis. In this situation I didn’t, and still don’t, see any way forward less drastic than a call for withdrawal…”

I really appreciate all the work you’ve done here assessing the paper, and you have me broadly convinced. But this is incredibly misleading and unfair. Of the four issues you allude to, you had only identified two when Bruya wrote the response “doubling down.” One you identified only two days before calling for retraction, and one you identified *after*. One alternative way forward would have been giving him more than two days to figure out what he should do with his paper—especially since you think he may not be that fluent with statistics.

I posted more about my general concern here below, but I lost track of this and I think it’s a serious enough misrepresentation to deserve being flagged. I know you’re likely experiencing this as just accumulating problem after problem (and the apparent delay between when Bruya wrote the response and when it was posted is confusing), but when exactly you raised the problems is of course central to Professor Bruya’s ability to respond.

And—congratulations on the little one! 🙂 Report

Jamie Dreier
Jamie Dreier
5 years ago

Jiminey,

Right, as that document says, “clear evidence that the findings are unreliable” whether due to misconduct or “honest error”. I was responding to this:

“Absent fraud, the scientific record should be clarified by a trail of correcting publications.”

I think you are now agreeing that fraud is not a necessary condition for warrant for retraction.

Of course, if there is no error, then it shouldn’t be retracted. But Prof. Wallace has given an argument that the paper has quite serious mistakes. So that’s an argument that the paper should be retracted.Report

C Marks
C Marks
5 years ago

In a peer-reviewed paper relying on quantitative methods, a claim of racism and sexism ought to make clear which of several possible senses were intended (individual; systemic but supervening on individual behaviors and attitudes and possibly other social objects; systemic but not supervening on individual behaviors and attitudes; an unintended outcome of certain methodological choices, and so on), and there should be some explanation and an appeal to quantitative data to support which of the senses was intended. Report

Jiminey
Jiminey
5 years ago

Indeed, Jamie, as in the example you posted where errors were found the authors themselves retracted to keep the progress of the scientific record clear. In this case, where there is a dispute about methods, a claim of error by a party would not ordinarily warrant retraction. Perhaps there will emerge some agreement as to some error, or maybe a dispute about proper methodology will continue in the literature. As I understand it, the sort of analysis involved is not a settled area and there are a variety of approaches. For this field to mature and be applied productively to the subject matter at hand, stifling dissent would be counterproductive, and the dispute should be played out scientifically until consensus emerges. Read through those guidelines in their entirety including their stated purposes, they are rather helpful. Report

David Wallace
David Wallace
Reply to  Jiminey
5 years ago

Jiminey says: “As I understand it, the sort of analysis involved is not a settled area and there are a variety of approaches.”

That sort of view is exactly what is driving my agitation on this issue. The analysis involved is *absolutely* a settled area. What exactly are the “variety of approaches” here?

– whether strongly correlated variables can be used in a multiple regression without doing a significance check? (This is 1930s mathematics.)
– whether you have to make your key methodological assumptions clear?
– whether, when you say something is very different from what would be expected by chance, you have to work out what *would* be expected by chance?

Report

Thom Brooks
5 years ago

I hope Metaphilosophy will seriously consider publish critiques of this paper. I continue to find the PGR enormously useful. Report

Jamie Dreier
Jamie Dreier
5 years ago

Jiminey,

Yes, it’s definitely better if the author retracts.

What “dispute about methods” do you have in mind? Prof. Wallace argues that several pieces of data analysis are wrong; scientific papers are frequently retracted because there are errors in analysis. What “sort of analysis” are you thinking of when you say “the sort of analysis involved is not a settled area and there are a variety of approaches”? Do you have in mind the way that Bruya reclassified sub-fields? Or the mathematics of multicollinearity in the regression analysis? Or what?

When papers are retracted because the editor is convinced they contain serious errors, it is not helpful to call this “stifling dissent”. That’s the sort of line taken by people who are sure that measles vaccinations cause autism and see the retraction of misleading papers as some kind of persecution. (I do realize that no children will get sick as a result of anyone’s mistakes in studying philosophy department status, so retraction does not have quite the urgency!)

I don’t see anything to disagree with in the document you linked to. In particular, the guidelines say, “Publications should be retracted as soon as possible after the journal editor is convinced that the publication is
seriously flawed and misleading”, which I think is right and general practice. Note that it doesn’t say anything about whether the issue is disputed; that’s not relevant.Report

Anonadjunct
Anonadjunct
5 years ago

Justin,

It’s not that you posted about the number of PhDs shortly after the data came out and Leiter posted. It’s that your post and your guest posts all tow the same line which just happens to be the opposite of Leiter’s views. Ditto for the posts attacking the pgr, and Lebon’s post. You can say it’s not a feud, but It sure looks like one. I am sure you and Leiter agree on some issues. I can think of one instance in particular where you were in rough agreement that happened in early November. However, I notice there is no mention of that rather public agreement on this blog.Report

Mike Otsuka
5 years ago

Wallace: You write, in your latest version: ‘Indeed, the most severe criticisms in this note– the silence, in the main part of the paper, about the reclassification of the PGR “Science” category as M&E, and the similar silence (not explained anywhere except in a graph label) about the definition of an evaluator being “from” an institution – would in other contexts be troublingly close to academic malpractice.’

Could you mention the contexts you have in mind? The most familiar context for academic malpractice is, I think, that of university disciplinary procedures for such things as plagiarism or fabrication of data (including for research conducted while an employee). While I agree that you’ve identified serious flaws with this article, what you mention above doesn’t strike me as coming close to warranting such disciplinary procedures.Report

Mike Otsuka
Reply to  Mike Otsuka
5 years ago

PS: By ‘such disciplinary procedures’, I don’t mean for plagiarism or fabrication of data in particular. Rather, I mean, more generally, disciplinary procedures by a university for misconduct by an employee or a student that plausibly falls under the umbrella of ‘academic malpractice’.Report

David Wallace
David Wallace
5 years ago

Otsuka: That may have been less clear than I’d intended. What I meant (and tried to clarify in the paragraph that followed) is that if someone with professional training in statistical methods – such as someone with a PhD in, and research-active in, a statistics-based empirical science or social science – it would have been fairly implausible for them to fail to notice that they had obscured central points of their methodology in a way that made their analysis appear much more persuasive than it would be if the methodology were clear, and so *in that context*, i.e. the context of work written by someone with that professional background, deliberate deceit would have to be considered.

As I have been at pains to stress, I *do not* make that accusation of Prof. Bruya. So far as I can see he does not have that sort of professional training, or that kind of research background (and, I assume, neither did Metaphilosophy’s reviewer). I am entirely happy to accept that these are honest mistakes (but for reasons I explain in my reply to Minh Nguyen above I think they’re still severe enough to make withdrawal of the paper the right way forward).

If I wished even implicitly to accuse Prof. Bruya of any wrongdoing I would not choose to do it in this medium.

Report

Mike Otsuka
Reply to  David Wallace
5 years ago

Thanks, David, for that clarification. Since I see that you’re still updating your document, perhaps a further update that incorporates these clarifications would be good.

In case this is relevant, I think academic malpractice that attracts disciplinary sanctions includes cases of negligence as well as deliberate deceit. A charge of plagiarism might be upheld even if it was (as those accused often claim) unintentional but rather a product of extreme carelessness.Report

WP
WP
5 years ago

I share the concerns about the call for retraction. When Professor Bruya wrote his response, Professor Wallace’s criticisms, as I understand them, were:

1) relegating discussion of the reclassified areas to the appendix made the body of the text significantly misleading,
2) we should consider a difference of four places (the average difference between PGR rank and Bruya “aggregated rank”) trivial, not “quite large,” given that the rankings have 50 places total, [at least as presented, this is an evaluative claim, not one about accepted standards in statistics]
3) Bruya suggests that the spread of PGR evaluators should target the APA’s division of specialties and it shouldn’t,
4) Bruya’s concerns about differing mean in specialty areas don’t adequately take alternative explanations into account.

This do not seem like enough to warrant retraction. The misleadingness in (1) could be *completely* solved by a correction rather than a retraction. Disagreement about whether the reclassification, as Prof. Bruya thinks, better gets at important methodological differences, as well as the criticisms in (2)–(4), seem like fairly substantive disagreements that would be appropriately addressed in followup literature. (Prof. Leiter, for example, does not seem to think a difference of 4 places is trivial at all and often posts explanations for departments “jumping” around that amount in the rankings.)

Prof. Wallace’s letter to Metaphilosophy highlighted, as the basis for retraction, (1) and the misleading classification of where evaluators were “from.” Prof. Bruya had not yet responded to the latter criticism and was given little chance to: Prof. Wallace identified it only two days (!) before. Not giving him a decent chance to reassess his own paper before going over his head to call for a retraction strikes me as pretty objectionable—especially when a prominent voice in the profession is already making completely unsupported allegations of fraud (speaking of per se defamation…).

I agree that we should adhere to the same standards as the sciences when it comes to quantitative analysis. It seems like we should also adhere to (at minimum) the same standards when it comes to giving our colleagues opportunity to respond to possible cause for retraction in their work. Is two days (much less two days at the end of a term and beginning of the holiday season) considered adequate time to give an author to respond to a serious criticism in the sciences? Is it considered appropriate to call for a retraction before your own analysis of the paper in question is complete? Report

David Wallace
David Wallace
5 years ago

@WP (replying to both your recent comments)

On your concern about “doubling down”: yes, that’s fair. I had intended to (i) give an illustrative list of my more severe concerns; (ii) register my concern in how Prof. Bruya had responded to methodological criticism. But you’re right that one natural reading of my post is that Prof. Bruya was “doubling down” in his defence of those very criticisms. That wasn’t intentional; apologies. (You’re right that the order of posts – with Prof. Bruya’s response published after my second set of criticisms but clearly written with knowledge only of my first – facilitates confusion.)

On the broader issue of timing (shared by Prof. Nguyen): I think there is a reasonable case for saying that this should have gone more slowly – and it’s certainly true that the rate of blog conversations on a subject induce a temptation to haste. However, on reflection I’d still have proceeded as I did. The issue is not so much that Prof. Bruya did not have time to reply; it’s that he did in fact reply, in some detail, and (i) his reply was very clear that he wasn’t even prima facie willing to consider methodological criticism of this severity as grounds for withdrawal; (ii) he made explicit that he would not be engaging in further discussion via informal means, and that critics should submit their own papers; (iii) his repeated defence of his methodology on the grounds that the PGR’s methodology is (he claims) just as bad made me pretty pessimistic about further informal conversations being productive even if (contra (ii) they were possible).

In that circumstance my judgement was that I had little prospect of getting a satisfactory resolution short of contacting the journal (in which case I might as well do it as soon as possible). That judgement may be wrong, and I can respect the view that it might be wrong. I would want to stress that (as I’ve explained in more detail above) I really don’t think this can just be annexed to philosophy’s usual objection-and-reply methods. I will try to resist the temptation to defend it further; I think I’d be adding epicycles, and (given the constraints of blog comments) running the risk of making some comment about Prof. Bruya with inferences I don’t intend, as with the above. (First-order discussions of methodology are one thing; second-order discussions of people’s intentions and reasonableness given those first-order discussions are the sort of fraught conversation I try to stay out of as much as I can help.)

On two specific things:

1) Should I have finished my analysis before complaining formally? I thought I had finished, twice. When I published my first set of concerns, one strong response I got (from “Nameless Graduate”) was that this was all a distraction from the real problem, which was the correlation between evaluators’ institution and institution grade, and the dominance of a small number of institutions; so I looked more carefully at that part of the paper and discovered very serious flaws there. Following that, one of Prof. Bruya’s responses to me was that I’d engaged with only a proper subset of his statistical analyses and that the rest therefore stood unchallenged, so I went to look at the remaining ones I hadn’t discussed and found the regression problem. (This is all illustrative of why statistics-based papers need careful expert peer review: in both cases, on surface reading everything looked sensible. Having said that, I’m not sure even expert peer review would have picked up the problem with institutional affiliation – I only spotted it when I tried to reproduce the analysis and got very different numbers.)

2) You and others are very kindly congratulating me on a new baby. What I actually said was that I have a new baby due in the next few days. That’s still true; congratulations are at the moment premature! (I mentioned it mostly because I’m probably going to drop rather sharply out of these conversations at some soon-but-undetermined point.)Report

WP
WP
Reply to  David Wallace
5 years ago

I share your frustration with Prof. Bruya’s response, but (as I think you agree) I think your later criticisms are among the strongest, and I see why he might have read you as having a Leiter-defending agenda, since I think your first criticisms do at points significantly misrepresent him.

For example, I’m sure what grounds the charge that Bruya treats the APA specialty breakdown as though it reflects demographics. As you say, the PGR categories aren’t intended to reflect demographics. So when Bruya suggests the APA categories might be used instead, why would that treat the APA categories as though it reflects demographics?

You say: “Philosophy of physics, for instance, is clearly smaller than ethics, but the PGR treats each as a special case because for prospective graduate students (the clearly-identified target of the speciality ratings), if they want to specialise in philosophy of physics they have fairly bespoke faculty needs. That carries no implication that philosophy of physics is comparable in size to ethics!”

I take Bruya’s actual argument to depend on what you say he ignores: The purported goal of the specialty rankings is to provide guidance for the different areas that prospective grad students might want to pursue. The APA’s specialty breakdown plausibly reflects the specialized areas of research that exist in US philosophy, so if the PGR is trying to provide specialty rankings for the areas students might be interested in, we could reasonably expect them to look something like the APA specialty groupings. Instead, certain kinds of APA-recognized specialties are systematically excluded. Rankings are offered for a quite fine grained breakdown of M&E and Science specialties, but for few non-Western or what Leiter calls “party line continentalist” specialties.

Of course some decisions have to be made about what’s too rare to be worth the effort, but, as I think you’ve noted, there is a special value to having rankings in relatively rare specialties—those are the ones that professors are least likely to be able to advise their students in. And I don’t think it’s true that there are more prospective grad students interested in, say, philosophy of social science than in existentialism, phenomenology, Indian philosophy, etc. The fact that Prof. Leiter might be pretty disconnected from the latter areas doesn’t seem like much a reason to exclude them, since the whole point is to get expert opinions. So why aren’t they included? (I think we all know the answer on the “party line continentalist” front: because Leiter thinks very poorly of the people who work in these areas, who also have very different ideas about what schools are best, and Leiter does not want them tainting the general rankings. For better or for worse, Leiter baked their exclusion into the PGR when he chose which specialties to rank.)

Anyway, I don’t mean to drag you into a neverending exchange. I just want to say that I see why Prof. Bruya was inclined to respond defensively, and I think more time might have helped.

If congratulations are no longer premature—congratulations! Either way, have a good holiday. 🙂 Report