Philosophers: Disappointingly Normal


Philosophers are sometimes thought of as expert thinkers, more rational and less prone to errors in reasoning than others. Whether this is true, though, is an empirical question, and it is one that several researchers have taken up over the past decade or so. Their findings regularly show that philosophers, like the ordinary folk, are susceptible to various cognitive biases. A recent study by Eric Schwitzgebel (Riverside) and Fiery Cushman (Harvard) hammers home the point, but in a way that even the authors admit to finding surprising. It’s called “Philosophers’ Biased Judgments Persist Despite Training, Expertise and Reflection.” Ouch. From the abstract:

We examined the effects of framing and order of presentation on professional philosophers’ judgments about a moral puzzle case (the “trolley problem”) and a version of the Tversky & Kahneman “Asian disease” scenario. Professional philosophers exhibited substantial framing effects and order effects, and were no less subject to such effects than was a comparison group of non-philosopher academic participants. Framing and order effects were not reduced by a forced delay during which participants were encouraged to consider “different variants of the scenario or different ways of describing the case”. Nor were framing and order effects lower among participants reporting familiarity with the trolley problem or with loss-aversion framing effects, nor among those reporting having had a stable opinion on the issues before participating the experiment, nor among those reporting expertise on the very issues in question. Thus, for these scenario types, neither framing effects nor order effects appear to be reduced even by high levels of academic expertise.

In short, there was an “across-the-board failure to find evidence for philosophical expertise and reflection in moderating biased moral judgment.”

What to do about findings like these? Well, if philosophers wish for “philosopher” to truthfully refer to something like “expert thinker” then perhaps we need to revamp certain parts of our philosophical training. Logic has been the core element here, but over time it has been supplemented with aspects of decision theory and statistical reasoning. We may need to do more here, not only educating philosophers about cognitive biases (as that itself seems to make little difference), but also training them in various debiasing techniques.

Going even further, since it has been suggested that “ambient and contextual factors may create high risk situations that dispose decision makers to particular biases” and that “fatigue, sleep deprivation and cognitive overload appear to be important determinants,” perhaps certain life management skills should be thought of as part and parcel of philosophical training. Maybe the ancients were onto something.

UPDATE: An article in Harvard Business Review (get over it) describes several strategies for overcoming biases in business contexts: make three estimates, think twice, use premortems, take an outside view, seek advice on objectives, cycle through objectives, use joint evaluation, and vanishing options test. Are any of these applicable in philosophical contexts? Are there analogues to these for philosophical problems? I can recall one paper that suggests a specific debiasing strategy in moral philosophy—“The Reversal Test: Eliminating Status Quo Bias in Applied Ethics” by Nick Bostrom and Toby Ord. Are there others out there? I mean besides basic stuff like Chapter 2 of On Liberty.

guest
14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
JH
JH
5 years ago

You can find several years’ worth of discussion of the general topic here:
http://www.overcomingbias.com/

I’ve done the Wason “2,4,6” test of confirmation bias on many philosophers professors. They all fail.Report

David Wallace
David Wallace
5 years ago

This is striking, but its significance shouldn’t be overstated. On (quick) reading of the experimental setup, it looks as if the “forced delay during which participants were encouraged to consider ‘different variants of the scenario or different ways of describing the case'” was about fifteen seconds. I don’t find it astonishing (I do find it a bit surprising) that professional training doesn’t much help in factoring out order effects in decisions made on that kind of fast timescale. It would be much more alarming if it were to turn out that ordering effects matter even if participants are given a couple of days to think about their answer and have to back it up with a 2,000-word argument.

This is in no way a criticism of the survey. But it’s compatible with something I find antecedently plausible anyway: that the skills of logical analysis that you learn in professional academic life are things that have to be deployed deliberatively over a relatively long timescale, and don’t help that much with swift judgements.

I also wouldn’t be *horrified* (I’d be fairly concerned) if it turned out that ordering effects persist even in my much longer-timescale study. Philosophy, like other (modern) academic disciplines, is a collaborative enterprise: it’s peer review and the cycle of criticism and response, not the logical acumen of the individual, that are the strongest constraint on irrationality in the subject. (In other words, philosoph*y* can be a lot more rational than philosoph*ers*, just as science can proceed fairly rationally even given the failings of individual scient*ists*.)Report

Matt Weiner
5 years ago

I participated in this study (or a related one) and was fairly underwhelmed when I saw the reported results. Two points:

1. Most importantly, the survey questions had very little to do with what philosophers do with respect to these examples. The experiment involved rating various stock philosophical examples on Likert scales. My immediate reaction on seeing them was “Gosh, my considered view is that what you should do depends on lots of details that haven’t been specified here! Too bad that’s not an option.” So my answers didn’t reflect my considered view very well.

We wouldn’t judge a baseball player’s expertise by asking them questions like “Curveball on the outside corner: Must Take, Should Take, Maybe Take, Maybe Swing, Should Swing, Must Swing?” It’s not obvious to me that asking philosophers similar questions about philosophical examples is any more illuminating.

2. It’s also not obvious to me that there’s anything irrational about giving different answers depending on the order in which scenarios are presented. If the two examples presented were Push and Organ Donation, I’d expect Push to get lower scores if presented after Organ Donation, because one of the arguments that Push is wrong is that it’s like Organ Donation which is obviously wrong.Report

Shea
Shea
5 years ago

Some pretty massive assumptions are required to move from what the study shows (i.e. that philosophers’ moral judgments about (possibly underspecified) cases are subject to framing and ordering effects) to the far stronger claim that philosophers’ judgments about cases in general are subject to biases.

First there’s the complaint that Matt Weiner (post #3) voices. I think ordering effects are a sign of imperfect rationality. (Though I think a reasonable argument can be made to distinguish imperfect rationality from irrationality tout court.) If Matt is right about ordering in the push/organ donation cases, it would mean that the ordering effect serves to remind us of salient information from our background views. This is why time for reflection is important: we can’t immediately recall all of our background views and the implications they might have for a particular case in 15 seconds. Indeed, the very requirement that one make a snap judgment almost insures that one will use cognitive heuristics that will be subject to such effects. Adding an extra 15 seconds won’t help much.

And I’m not sure why anyone would expect philosophers to employ more reliable cognitive heuristics than anyone else. Professional training in philosophy develops one’s ability to reason in a slow, deliberate, and reflective manner. This provides a crucial disanalogy between philosophers and chicken-sexers and radiologists. Philosophers don’t make a living by making repetitive judgments about thousands of cases. Rather, they think long and hard about a handful of cases. We are not experts at making snap judgments: we are experts at slow and prodding deliberation.

I also expect judgments about moral cases to be subject to different psychological forces than, say, judgments about sleeping beauty or fake barn cases, in that the former will involve more emotion. I would also expect our moral judgments to be very sensitive to background social information. But this is because I expect these judgments to be an adaptation that helps individual human beings fit into society, rather than stemming from a cognitive faculty that developed to be receptive to a realm of eternal non-physical truths. But then again, the literature on sleeping beauty tends to be framed in terms of arguments stemming from general principles rather than by brute appeals to intuitions about the case. Though I would be interested to see if one can generate similar results for Gettier cases, etc. At any rate, I certainly wouldn’t infer from this study that all philosophers’ judgments are subject to the same effects. More importantly, I wouldn’t infer any strong methodological claims as it is unclear whether any philosophical research is based on the sort of snap judgments that the experiment tests, for the reasons I have outlined above.Report

Anon
Anon
5 years ago

It turns out philosophers are disappointingly *ab*normal. Normal people, when confronted with their biases, shrug, reluctantly admit them, then go back to business as usual. Philosophers will continue to insist they have none.Report

Clement Loo
Clement Loo
5 years ago

It strikes me that several criticisms above address claims that aren’t made by the study’s authors.

From my – admittedly quick and not particularly careful – reading it seems that their claim is simply that training doesn’t have an effect a couple of putative biases; insofar as that those who report expertise perform similarly on a task intended to reveal those biases than those who do not report expertise. I think the real question – one that Matt suggests above – is whether the task used by Schwitzgebel and Cushman actually measures whether one is subject to ordering or framing effects.

I suppose one could argue about whether the questions used in the study actually capture ordering or framing effects. But, since -as far I saw – they never listed their questions, whether those questions are adequate or not is a matter of speculation for all of us who didn’t participate in the study.Report

Shea
Shea
5 years ago

I think the x-phi studies done on things like cross-cultural variation among intuitions are important and have methodological implications. The reason for the difference is that one would expect ordering and framing effects to eventually filter out as philosophers consider cases in greater detail and interact with one another, whereas one would expect cultural biases to reinforce themselves when they are shared within an intellectual community. To be clear: I am not dismissing the results of the experiment. They are fairly robust. It’s just that they don’t have immediate broader implications for philosophical methodology.Report

Rob H
Rob H
5 years ago

As it says in the linked paper, the exact texts used in the study are found in the supplement (which you can get from Eric Schwitzgebel’s webpage). The authors used boiler plate trolley problems and framing problems. I sincerely doubt one can make a strong argument against the study based on textual problems or screwing up the presentations. Additionally, assuming that being a Professor/PhD with certain AOS is a good proxy for “Philosopher-who-has-considered-trolley-cases-in-great-detail-and-engaged-in-conversation-with-other-experts (via research, publications, and conferences)”, then the empirical evidence supplied in the paper suggests order/framing effects do NOT filter out in the long run.
Pardon any typos….sent form my phoneReport

Matt Weiner
5 years ago

[email protected]: “Normal people, when confronted with their biases, shrug, reluctantly admit them, then go back to business as usual.”

Assumes facts not in evidence, as they say.

I realize I wasn’t clear about what I said about the ordering effect. I think that, even if the subject group were entirely rational, we would expect people to be more reluctant to Push when the Push question is presented immediately after a (decent) argument against Pushing then when it is presented after the argument, because the second group hasn’t seen the argument when they’re answering the question. And I think the Organ Donation example is an enthymematic argument against Pushing, since most people will consider it bad and it’s not too difficult to see the ways in which it parallels Push. So I would expect to see ordering effects with Organ Donation and Push.

I also think, though this is perhaps more controversial, that Push and Switch are similar–Push can be seen as an enthymematic argument against Switching, so ordering effects can be rational.

Perhaps one might argue that these considerations should not apply to philosophers, who’ve already seen both cases umpteen times.Report

Shea
Shea
5 years ago

“Additionally, assuming that being a Professor/PhD with certain AOS is a good proxy for “Philosopher-who-has-considered-trolley-cases-in-great-detail-and-engaged-in-conversation-with-other-experts (via research, publications, and conferences)”, then the empirical evidence supplied in the paper suggests order/framing effects do NOT filter out in the long run.”

The evidence suggests that when presented with the cases philosophers are still just as likely to make snap judgments that are subject to framing/ordering effects as laypeople. I’m not denying that. As I said, we should not expect philosophers to be experts at making quick judgments. By “filter out in the long run” I mean that ordering and framing effects are unlikely to determine what positions are ultimately defended in the literature. This is because during the process of publishing a paper on the trolley problem a given philosopher is likely to consider the two cases multiple times in multiple orders, and if they do somehow appeal to something that depends upon an ordering or framing effect it is fairly likely that a blind reviewer or editor will pick up on it. The same goes for recognized framing effects. The sheer fact that we explicitly recognize ordering and framing effects to be performance errors means that we will tend to eliminate their effect on our positions in the long run. What would it mean to suggest otherwise? Is the claim that one can predict whether any philosopher treats switch and push cases equally based upon the order in which they were first introduced to the cases back in their undergrad intro to ethics class? That seems pretty unlikely, imho. Indeed, it would contradict the results of the experiment. The experiment would seem to suggest that the philosophical views about trolley cases varies constantly in real-time depending on what people are thinking about and simply cannot be read off of what is defended in the literature. Again, that seems unlikely. In contrast, we generally do not collectively recognize cultural biases, and thus do not collectively act to correct them. Rather, we tend to think they are universal until empirical work tells us otherwise. This is why experiments confirming the latter form of bias are more methodologically worrisome. Though to be clear, it is still depressing that philosophers tend to make these sorts of mistakes in their particular occurrent judgments.Report

n
n
5 years ago

Apropos the comment on the ancients being on to something, recall this award winning paper by Ann J. Cahill and Stephen Bloch-Shulman:

http://dailynous.com/2014/04/29/2014-lenssen-prize-winners/

Argumentation Step-By-Step: Learning Critical Thinking through Deliberate Practice

Abstract:
In this paper, we offer a method of teaching argumentation that consists of students working through a series of cumulative, progressive steps at their own individual pace—a method inspired by martial arts pedagogy. We ground the pedagogy in two key concepts from the scholarship of teaching and learning: “deliberate practice” and “deep approaches to learning.” The step-by-step method, as well as the challenges it presents, is explained in detail. We also suggest ways that this method might be adapted for other classes.Report

Vincenzo Politi
Vincenzo Politi
5 years ago

The first sentence should be “Philosophers always think they are expert thinkers, more rational and less prone to errors in reasoning than everybody else”.Report

PeteJ
5 years ago

I often see the ‘hard’ problem framed by philosophers of mind as, ‘How to explain how consciousness arises from brains’. I feel this tells us that the article is mostly correct. Such schoolboy errors are surprisingly common, and may account for the perceived hardness of many problems.Report

Anon
Anon
5 years ago

What a surprise! We philosophers are human, too!Report