Actually, contrary to what you say, it’s clear that your explanation of the data is on more solid footing than anothercommenter’s conjecture. The data show that, even when you control for AOS and year of graduation, being a woman increases your chances of getting a permanent job significantly. Anothercommenter conjectures that it’s because women are not taken seriously and, as a result, produce superior work while they are in graduate school.

First of all, given how large the effect of gender seems to be, it would presumably have to be the case that women’s work is a *lot* better on average than men’s work. That is a very strong claim which needs to be independently supported. Now, the only data we have that independently bear on that hypothesis, namely the average number of publications for each gender, suggests that the opposite is true.

You suggest that, for anothercommenter’s hypothesis to explain the data, women’s work doesn’t have to be a lot better to men’s work, because for any particular job there are often a lot of candidates that are judged to be equally qualified. The only evidence you give for that claim is that, for any particular job, there are a lot of qualified candidates. But, from the fact that there are a lot of qualified candidates, it doesn’t follow that a lot of those are *equally* qualified.

Moreover, even putting that aside, it’s clear that anothercommenter’s hypothesis is a bad explanation of the data. As DoubleA notes below, in order to explain the data, it’s not enough that women’s work be on average better, it has to be recognized as such. But anothercommenter’s explanation of the hypothesized superiority of women’s work is that their work is generally not recognized in the profession. So it would have to be the case either that people’s bias suddenly disappears when they become part of a hiring committee, or that women’s work is so much better than men’s work that it’s enough not only to overcome that bias, but even to make them significantly more likely to get a job after the bias has been overcome.

The hypothesis that employers have a preference for women, on the other hand, doesn’t have any of those problems. In particular, it coheres with other things we know beside the hiring patterns shown in Carolyn’s data, such as the fact that a lot of people in the profession and in academia more generally are on record saying that affirmative action for women is justified.

Finally, nothing I have said implies or even suggests that I am “angry at women” and, indeed, I am not. As I already said, what really bothers me is the bad faith in this debate, which I’m sure is obvious to everyone but a handful of true believers. It’s okay if you think that women *should* get preferential treatment in hiring. I suppose that reasonable people can disagree about that. But please don’t use fallacious arguments and make unsubstantiated claims to deny that, based on the data we have, the most plausible hypothesis is that women *do* get preferential treatment in hiring, because that’s just dishonest.

A few years ago, when people were talking about the underrepresentation of women in the profession, we were told that one of the explanations was that women were less likely than men to be hired even when they were equally qualified. But now we have data that clearly show that not to be the case and, all of sudden, all sorts of ad hoc hypotheses are floated to explain the data. It’s pretty clear that, for some people, no amount of data will ever be enough to convince them of the truth of something they don’t want to accept. That’s what is annoying me.

People have always been saying that women have to be better than men to get through graduate school. It may not explain the whole effect, but no, it’s definitely not ad hoc to bring it up now.

You’re putting a lot on anothercommenter’s particular description of the situation without considering whether there might be a better hypothesis in the same spirit, which would be normal in a less politically loaded inquiry.

I think the core of the claim is just: there are a bunch of cultural factors discouraging women from pursuing philosophy, and what makes women overcome those factors is often that they’re really good at it, enough so that women who self-select into philosophy are, on average, better than the men who do. There are also a bunch of factors that cause graduate school women to, in general, feel more of a need to prove themselves have less confidence in their work, which tends to result in stronger work.

It’s not that the value of women’s work is never recognized. It’s that women, especially in person, are less likely to be given the benefit of the doubt, less likely to be assumed to be saying something right, and so if there are ambiguities and unaddressed problems, they’re more likely to get pressed, which essentially amounts to a more rigorous graduate school experience. Because of this, women tend to spend more time per paper, and so they tend to write fewer, but better, papers. If hiring committees tend to prefer one excellent paper to two good papers (which I think is true), this would be an advantage.

Your participation here seems kind of disingenuous. “I very much doubt that, if the analysis had revealed a bias for men, anyone would have floated that kind of hypothesis to explain away the result.” Well… as you know, the plausibility of alternative explanations depends on your other evidence. Plenty of women have good reason to believe there are serious disadvantages to being a woman in philosophy.

By the way, I have also read Quine and Duhem, so I know that if you want to save a hypothesis you can always add epicycles to do it. I don’t see why that should give us any reason to doubt the hypothesis is false unless the epicycles are independently supported by evidence.

I’m going to ask if we have the data for the Philosophical Review, which is edited at Cornell, where I’m doing my PhD. I’ll let you know what I’m told and, if the data are available, what they say.

Just because most of the evidence produced to argue that women face a hostile environment in philosophy is anecdotal, it doesn’t mean that we can’t test the specific claims that are made. In fact, many of the claims anothercommenter made could be tested, some of them pretty easily. I already performed a quick and dirty test of the hypothesis that women are less likely to be invited to give talks or lectures and found it to be false. It would be relatively easy to perform that test more rigorously, but again I don’t see anyone else trying.

You’re right that I don’t regard anecdotal evidence very favorably, but I don’t think that should be controversial, because the social science literature is replete with examples that show how unreliable that kind of evidence is. That’s not to say that anecdotal evidence can’t play a useful role, but I think it’s a purely heuristic role, namely the role of suggesting hypotheses for which real evidence must then be sought. But, before we test a hypothesis, we have to make sure that it’s consistent with the data we already have, which I don’t think is the case with anothercommenter’s hypothesis.

As for the suggestion that we test that hypothesis by reviewing job dossiers blindly, I think everybody knows that there is exactly zero chance this is going to happen, so making that suggestion comes pretty close to saying that we’re not going to test it. Indeed, so far, I don’t see anyone beside me doing anything to uncover evidence. I just don’t see how I’m being unfair by pointing that out, especially given the history of dishonesty in that debate. Perhaps if you were one of the only people commenting about that issue under your own name and people accused you of being angry at women for absolutely no good reason you would see things differently.

I find the kind of mechanisms you mention a lot more plausible than the ones proposed by anothercommenter. I guess I wouldn’t be entirely surprised if, because of that kind of factors, women were other things being equal more likely to produce better work. But I don’t know if other things are equal and, in any case, what I find utterly implausible is that it could explain the magnitude of the effect of gender that we observe in the data collected by Carolyn and her team.

As I already pointed out, it’s a very large effect and, moreover, we have good reasons to think it’s underestimated since the regression didn’t control for the number of publications and the data we have about that suggest men have on average significantly more publications. In light of all that, if you told me that you believe that the hypothesis that women produce on average better work than men could explain the entirety of the effect of gender in hiring or even a large part of it, I would have a hard time believing you.

But I’m willing to play ball, if we can finally put that thing to rest. We seem to agree that a good way to test that hypothesis would be to look at acceptance rates broken down by gender in journals that practice blind review. I still haven’t heard back from the editorial manager of Philosophical Review, so I don’t know if we have that information, but I will be sure to keep you posted as soon as I hear back from her. But Philosophical Review is just one journal, although a prominent one, and I don’t know if they have that information. It’s possible that most journals don’t record that information.

It would be very useful if Justin could make a post asking editors of journals who practice blind review for that information if they have it and to encourage them to collect it if they don’t. I think a lot of them read Daily Nous, so I’m sure it could be effective. Would you join me in asking Justin to write such a post? I fear that unless we actually try to uncover the evidence we need to test anothercommenter’s hypothesis we are just going to waste our time in endless disputes.

I don’t have time to fully respond right now, but I came across this and thought you’d be interested: http://dailynous.com/2014/07/08/making-philosophy-journal-statistics-publicly-available/

I appreciate you commenting under your own name, and I get that it must suck to feel like you have to do that to get your perspective out there without being dismissed as a troll. Because I think this is an important conversation, I do want to say that your first response to anothercommenter (and the same line in comments after that) reads to me like it’s in bad faith and I understand why other commenters took away that you were angry at women. The use of anecdotal evidence (etc.) here seems to me to be completely normal and appropriate, and I think some of the standards you’re applying would be really unusual on any other topic.

Didn’t post this earlier because it’s been way too long since I’ve thought about statistics and I don’t feel confident about it, but I think that, if work quality among men and women was normally distributed with the same SD, the mean work quality for women would need to be a bit less than .4 standard deviations higher than the mean for men to fully explain this effect—at .4 and with the threshold for “highly qualified” at the mean for women, 50% of women and 34% of men would be “highly qualified,” odds ratio 1.94. But… it’s been a while.

« Didn’t post this earlier because it’s been way too long since I’ve thought about statistics and I don’t feel confident about it, but I think that, if work quality among men and women was normally distributed with the same SD, the mean work quality for women would need to be a bit less than .4 standard deviations higher than the mean for men to fully explain this effect—at .4 and with the threshold for “highly qualified” at the mean for women, 50% of women and 34% of men would be “highly qualified,” odds ratio 1.94. But… it’s been a while. »

I didn’t check your numbers, but off the top of my head, that sounds about right to me. However, I don’t know if we’re supposed to think that 0.4 standard deviation is a small effect, but it’s actually a pretty large one. I don’t think that you could find any study that shows a difference between men and women that even come close to such a magnitude for any cognitive ability or even for something like GPA. Moreover, in addition to the assumption that work quality is normally distributed (which is hardy obvious), your argument rest on the assumption that it has the same standard deviation among both men and women. (Normal distributions are pretty ubiquitous in nature, but that doesn’t mean that you should expect any particular distribution to be normal, unless you have good reasons to think so. Philosophy of science published a very good article by Aidan Lyon in 2013 in which he argued that the typical explanations for why distributions that are known to be normal are normal were not satisfactory.) However, if the standard deviation for men is larger than for women, even a mean difference of 0.4 standard deviation — which again is pretty large — would result in a smaller odds ratio. But there is plenty of evidence, as well as theoretical reasons to think (e. g. men only have one X chromosome), that men show more variability on cognitive abilities and a lot of other traits, so that’s hardly a trivial assumption.

Even if we accept all those assumptions, based on the only data we have so far (i. e. the ones published by Dialectica), such a difference between the mean quality of work by men and women is overwhelmingly unlikely, as I have pointed out above. The calculation I did above with the data from Dialectica using your assumptions should give you a sense of how large the kind of difference between the quality of the work of men and women would have to be in order to explain the hiring data. In fact, using the data from Dialectica makes it look a lot easier than it actually is, because over the entire period in those data the acceptance rate is unusually high for a top journal. I’m guessing that’s because in the first few years of the journal the overall acceptance rate was very high. But in the last 10 years it was about 8% and, in most top journals, it’s probably lower than 5%. Take a journal like Philosophers’ Imprint for which the acceptance rate was 4% last year according to their website. (It’s also a good choice because, since it’s an online journal with no space limitations, the way acceptance decision are made probably corresponds more closely to the simple model where anything that passes a certain threshold of quality gets published.) Let’s assume that the quality of work submitted to that journal is normally distributed with equal variance for both men and women and that only 4% of the work submitted by men passes the threshold for publication. Under those assumptions, if the quality of the work submitted by women is on average 0.4 standard deviation above that of the work submitted by men, we would expect the acceptance rate of women to be approximately 8.8%, namely more than twice the rate for men! How much would you bet that, if Philosophers’ Imprint published the acceptance rate broken down by gender, we would see anything close to that? My guess: not a lot. (But if I’m wrong please take the bet with me, I could definitely use the money.)

« Anyway, I’m most familiar with psychology, but something you see pretty often in psychology journals:
paper: I got results. Here’s an interpretation.
comment on paper: You haven’t ruled out this alternative interpretation, which is more consistent with what a lot of people believe. Evidence X may seem to suggest against this alternative interpretation, but there are reasons to doubt it.
reply to comment: o. Here’s some additional evidence. »

I agree that you see that often, but it also has little to do with what’s going on here. When in a scientific journal someone points out that an alternative explanation H that is more consistent with X, where X is something a lot of people in the field believe, has not been ruled out, X is not just anything that people happen to believe. Rather, X is something that is taken to be well supported by evidence, not anecdotal evidence but serious evidence. For instance, X is a popular theory, which has been successfully tested in a variety of ways. It doesn’t matter if a lot of people believe something unless they have good reasons to do so. There are plenty of examples of things which a lot of people, including a lot of philosophers, believe even though they are false. So perhaps a lot of people believe that women face a hostile environment in philosophy, but I can’t say that I’ve seen a lot of non-anecdotal evidence to that effect. (Warning: this was an understatement.) When I performed a quick and dirty test of one precise claim made by anothercommenter, namely the claim that women are less likely to be invited to give talks or lectures, I found it to be false. If people want to test that hypothesis more rigorously, they are welcome to do so and share the results with us.

Moreover, it also has to be the case that we have good reasons to think that H really is more consistent with X, which again requires more than mere speculation. One can speculate that, if women face a hostile environment in philosophy, it makes their work better. One can probably even find a few studies that lend some very indirect support to that claim. But then again one can also speculate that, if women face a hostile environment in philosophy, it negatively impacts the quality of their work. And one can surely find studies that lend some very indirect support to that claim. Scientists usually require that the connection between X and H be less ambiguous in order to consider it a serious problem that H was not ruled out. It’s not enough that you be able to cook up a somewhat plausible story to the effect that X makes H plausible, especially when you can just as easily cook up an equally plausible story to the effect that X makes H implausible. That’s even more true when H, as the hypothesis that women on average produce better work, is actually at odds with the only data we have that directly bear on it.

Finally, I don’t think a lot of scientists would take seriously the complaint that H has not been ruled out, unless we have good reasons that H could explain the data. But, as the back-of-the-envelope calculations I did above suggest, given the magnitude of the effect of gender in Carolyn’s data, it’s rather unlikely that the hypothesis that women produce on average better work could explain it. Again, if one really wants to deny that something is true, there is always a way to do so.

Anyway, I feel that we’re past the point where this discussion is likely to remain productive, so I’ll just let you have the last word if you want it. I’ll just summarize once more what I take to be the situation. Anothercommenter suggested that the hypothesis that women produce on average better work than men could explain the fact that women are more likely to find a permanent academic position even if employers didn’t have any preference for women. I think that hypothesis is no more plausible than the contrary hypothesis that men produce on average better work than women and that neither of them is particularly likely. The only evidence we have for the moment that bear directly to anothercommenter’s hypothesis, namely Carolyn’s data about number of publications and Dialectica’s data about acceptance rates broken down by gender, clearly speaks against it. Moreover, a few back-of-the-envelope calculations suggest that even if that hypothesis is true, it’s very unlikely to explain the magnitude of the effect of gender in the data or even to come close to that. If people still think that’s a promising avenue to refute the hypothesis that employers have a preference for women, I’m really not sure what else I could tell them that would change their mind.

As for the question of who is arguing in good faith in this debate and who isn’t, I will let the people who read this make up their own mind. If they care to do so, I invite them to consider whether I have ever changed my standards since I first talked about that issue back in 2014, how often I had to change my hypothesis and add epicycles to it in order to accommodate inconvenient evidence, whether I’m commenting under my own name and therefore taking responsibility for everything I say, how many times I took the time to seriously engage with the data and perform statistical analyses on it, etc. Then, if they want, they can ask themselves the same questions about the people I have been up against. I can’t say that I’m overly worried about what lesson most people will draw from that comparison.