Crafting a Critical Thinking Course that Sticks with Students (guest post)
A forthcoming study shows that a critical thinking course focused on a few good, relatively easy to learn, and useful reasoning strategies can impart lessons that remain effective long after the course has ended.
The study, authored by Michael Bishop (Florida State), Adam Feltz (Oklahoma), and Paul Conway (Southampton), will be coming out in The Journal of Experimental Psychology: Applied.
In the following guest post, Professor Bishop describes the course, the motivation for it, the design of the study, and the results.
An appendix at the end of the post contains the study’s questions.
Crafting a Critical Thinking Course that Sticks with Students
by Michael Bishop
Two large longitudinal studies conducted by Adam Feltz, Paul Conway, and me suggest that a critical thinking class can promote large and enduring improvements in how students think about important everyday issues. These studies are described in an article forthcoming in The Journal of Experimental Psychology: Applied.
Background
In Epistemology and the Psychology of Human Judgment (2005), JD Trout and I defended the view that good reasoning strategies are reliable, cheap (i.e., easy to use), and useful. We argued that this view, Strategic Reliabilism, provides a powerful framework for evaluating and repairing our epistemic practices.
To prove the ameliorative power of Strategic Reliabilism, I revised my critical thinking class. This turned out to be a long-term project. (Thanks, tenure!) I stopped teaching reasoning strategies that were expensive (e.g., expected utility, rules of logic and probability) or of dubious utility (e.g., informal fallacies). And I started teaching good, cheap, and useful reasoning strategies. For the purposes of a critical thinking class, I took good reasoning strategies to be those that tend to produce accurate judgments (for theoretical reasoning) or favorable outcomes given the reasoner’s values (for practical reasoning). The class teaches students to use reasoning strategies for sampling arguments, testimony, the law of large numbers, herd reasoning, overconfidence, mental accounting, confirmation biases, and status quo biases. Here are five examples of good, cheap, and useful reasoning strategies I teach, each with its own rhyming ditty.
Causal Rule: Don’t infer causation from correlation. Causal arguments have holes if they don’t have controls.
Regression Rule: For events that have a central tendency, recognize that extreme events tend to be followed by more average or typical events. If something impresses, it probably regresses.
Sunk Cost Rule: Unrecoverable costs cannot be a consideration that favors one course of action over another. Wasted money and time will vex if you let ‘em, but when making decisions it’s best to forget ‘em.
Opportunity Cost Rule: When reflecting on a course of action, recognize the cost of not pursuing the best alternative. If an option’s imperative, you have reasons comparative.
Gambler’s Fallacy Rule: Treat statistically independent events as statistically independent. You should guess runs regress but doubt they even out.
Do students learn to think better when they learn to use good, cheap, and useful reasoning strategies?
Study Design
113 critical thinking students (the training group) and 284 students in other introductory philosophy classes (the control group) took a survey at the start and end of the semester. The survey asked them to evaluate 5 instances of reasoning, each of which violated one of the 5 rules described above. We waited 16 months, and we gave them a survey that asked them to evaluate different instances of reasoning, each of which violated one of the 5 rules.
The surveys included two skepticism checks, reasoning that did not violate any rules taught during training. This was to assess whether training had simply taught students to be hyper-critical. (See Appendix for all 14 instances of reasoning the students evaluated.)
Results
Training improved reasoning on 4 of the 5 rules, with the gambler’s fallacy being the lone exception. On the pre-test, both groups positively evaluated reasoning that violated the causal rule, the regression rule, the sunk cost rule, and the opportunity cost rule. On the post-test and 16-month delayed test, the training group was significantly more critical. These improvements were large. Averaging across all 4 items, on the post-test, the training group’s mean was about 1.4 standard deviations better (i.e., more critical) than the control group’s mean. And it was almost 1 standard deviation better on the 16-month delayed test.
On the pre-test, both groups evaluated the skepticism checks positively. On the post-test and 16-month delayed test, the training group was somewhat more critical than the control group, although the difference was not statistically significant. Did training promote unwarranted skepticism? Or wariness of reasoning proposed by their critical thinking teacher? Or a healthy dose of epistemic humility? This issue requires more attention. (See Appendix for more detailed results.)
3-Month Study
The previous year, we conducted a similar study with 395 students and two delayed tests. The first occurred 3 months after the class ended and was identical to the pre- and post-tests. The second occurred two weeks later and included all new reasoning items. The results were similar to the results of the 16-month study. On the post-test and delayed tests, the training group showed large improvements on 4 of the 5 rules, with the gambler’s fallacy being the exception. The response rates on the 3-month delayed tests were 39.2% and 32.9%, whereas the response rate on the 16-month delayed test was 23.7%. We speculate that we lost more students to graduation on the 16-month study.
* * *
Appendix: 16-Month Study Details
On each survey, students were given 7 instances of reasoning (5 rule-violation items and 2 non-rule violation items). They were asked to accept the premises and then evaluate on a 7-point scale whether the premises give us “no reason” (1), “moderate reason” (4), or “very strong reason” (7) to accept the conclusion. The bar graphs (with the lowest possible “no reason” response as the zero point) give the mean rating on each question. (Grey = control group. Black = training group.) All measures, datasets, and analyses for the 3-month and 16-month studies are on this project’s OSF page: https://osf.io/syxr9/.
Causal Rule
Pre & Post Tests: The Anti-Crime Law reformed Moldova’s social and judicial policies. The goal of the law was to reduce crime. In the two years following the enactment of the law, the crime rate in Moldova dropped by 3%. Conclusion: The Moldovan Anti-Crime Law was an effective deterrent: It reduced crime.
16-Month Delayed Test: The state of Caledonia puts in place new water pollution regulations. One year later, water pollution levels in Caledonia are unchanged. The lakes and streams of Caledonia are just as polluted as they were when the new regulations were put in place. Conclusion: The new water pollution regulations had no effect on Caledonia’s levels of water pollution.
Regression Rule
Pre & Post Tests: Sam is a good baseball player. But sometimes he has a bad game. After he has a bad game, Sam does extra mental exercises before his next game. And his performance almost always improves. Conclusion: Sam’s extra mental exercises have a good effect on his performance.
16-Month Delayed Test: Pat is a swim coach. Pat rewards swimmers who have a great race with a “Great Race” award. After swimmers receive a “Great Race” award, they usually swim slower in their next race. Conclusion: Pat’s “Great Race” award tends to have a negative effect on swimmers’ performance.
Opportunity Cost Rule
Pre & Post Tests: If you sign this contract with Corporation XYZ, your costs over the next year will be $50,000 more than your revenues. Conclusion: You should not sign this contract with Corporation XYZ.
16-Month Delayed Test: The Cannington Corporation is deciding on an investment strategy for the next three years. If they adopt the “whiplash” strategy, their investment portfolio will grow by 8% each year for the next three years. Conclusion: The Cannington Corporation should adopt the “whiplash” investment strategy for the next three years.
Sunk Cost Rule
Pre & Post Tests: Sam is financially comfortable but not wealthy, and he takes a one-week vacation every year. Three months ago, he paid a $500 deposit to rent a beach house during his vacation. He now realizes that he would enjoy a week at a mountain resort far more than at the beach house. The $500 deposit on the beach house is non-refundable. Conclusion: The $500 deposit is a factor that weighs in favor of going to the beach.
16-Month Delayed Test: Freedonia has been at war with its neighbor Sylvania for four years. They’re fighting over Pearl Island and its rich natural resources, which Sylvania currently controls. In this war, Freedonia has paid a very heavy price in terms of lost lives and money, and it must now decide whether to continue to fight or sign a peace treaty. If Freedonia continues to fight, it will eventually defeat Sylvania and gain control of Pearl Island. If Freedonia signs a peace treaty, Sylvania will stay in control of Pearl Island. Conclusion: The cost in lives and money Freedonia has paid over the past four years is a factor that weighs in favor of continuing the war.
Gambler’s Fallacy Rule
Pre & Post Tests: A coin is fair. That means that the coin has a tendency to come up heads half (50%) of the time and tails half (50%) of the time. So far, we’ve flipped the coin 20 times. By chance, it has comes up heads 15 times and tails 5 times (75% heads). We plan to flip the coin 80 more times (for a total of 100 flips). Conclusion: In the next 80 flips, the coin is likely to come up tails more often than heads.
16-Month Delayed Test: A spinning wheel is fair. That means that on any spin, the wheel has a tendency to come up green half (50%) of the time, and red half (50%) of the time. So far, we’ve spun the wheel 20 times. By chance, it has comes up red 15 times (75% red). Conclusion: In the next series of spins, the wheel is likely to come up green more often than red until the proportion of green and red outcomes are closer to even (50-50).
Skepticism Check (Causal Reasoning)
Pre & Post Tests: The Anti-Crime Law reformed Moldova’s social and judicial policies. The goal of the law was to reduce crime. In the two years following the enactment of the law, the crime rate in Moldova dropped by 3%. In similar countries that did not pass such a law, the crime rate dropped by 1%. Conclusion: The Moldovan Anti-Crime Law was an effective deterrent: It reduced crime.
16-Month Delayed Test: 23 cities, including Townville, imposed a ban on high cholesterol oils in restaurants and grocery stores. Since these bans were imposed five years ago, health measures (such as levels of heart disease, obesity, diabetes, and mortality) in those cities, including Townville, have remained constant. But over those same five years, in similar cities that have not banned high cholesterol oils, those health measures have gotten noticeably worse. Conclusion: The high cholesterol oil ban had a positive effect on the health of the Townville residents.
Skepticism Check (Practical Reasoning)
Pre & Post Tests: If you sign this contract with Corporation XYZ, your costs over the next year will be $50,000 more than your revenues. Your best alternative – signing a different contract with Denko Corporation – will result in a significant profit (your revenues will be considerably greater than your costs). Conclusion: You should not sign this contract with Corporation XYZ.
16-Month Delayed Test: Ecuador is deciding whether to implement a new health care policy. They’re considering the best three options: Plan A, Plan B, and Plan C. The three plans will cost about the same. (Health care costs will make up about 8.4% of the economy on all three plans.) But Plan A has far better health outcomes for the people of Ecuador than Plans B and C. Conclusion: Ecuador should adopt Plan A.
Discussion welcome.
I wonder if the survey could be shared so that our students could also take it?
It’s so hard to design something that plausibly measures learning like this. I’ve tried before (and basically the pre-test was the final exam questions and an analysis of reasoning in an essay). Thanks for thinking this through so thoroughly and doing this work.
The heart of the survey is in the Appendix of this post. Students evaluated 7 instances of reasoning in the pre- and post-tests, and 7 different instances of reasoning in the 16-month delayed test.
If you’d like to see the exact instructions and the exact survey questions, they appear in the Appendix of the full manuscript, which I believe you can download here: https://philpapers.org/rec/BISCTC
Oh thank you so much. I’d love to use those if that is OK? Noting authorship at the top of the handouts?
Please feel free to use the surveys as you’d like.
Some educational psychologists say that critical thinking cannot be taught in abstraction from the subject matter of a class, or at least, that’s the way it is for ‘novice’ learners. So for example, if you teach your students ‘critical thinking’ (the subject) by teaching them when to spot the sunk cost fallacy, that doesn’t mean they’ll be able to engage in critical thinking (the ability) when they think about, say, free will (a subject often taught in intro). The converse is also true, of course.
Is it really surprising, then, that a bunch of intro phil students did worse on a ‘critical thinking’ (the subject) test than students who already took that class? Look at it like this. Don’t you think the ‘critical thinking’ students would do worse on an intro phil exam than the ‘critical thinking’ students? If so, it’s hard to see how any of this shows students improved on critical thinking (the ability and the focus of Bishop and Trout’s book), rather than knowledge of ‘critical thinking’ (the subject).
The lack of statistically significant improvement in the skepticism checks (rules not taught in the critical thinking class) looks like confirmation of this supposition.
I have been thinking about this topic too recently. I’ve found this website helpful as a starting point for some citations to relevant work on the topic.
I wondered something similar. I was expecting that they would randomize how they teach critical thinking in two different classes… (“traditional way” with fallacies, logic and probability” and then their way “Strategic Reliabilism”)… but maybe they are pursuing such studies in their follow up work?
Even then, it would only show that they learned the subject of fun slogan critical thinking better than the subject of boring logic critical thinking. It would not show that they have improved in their critical thinking abilities as marketed by the authors (i.e.,the kind that can be applied across subject matters).
The authors made a good move by comparing different students in different subject matters. But they made a bad move (all due respect/I don’t have tenure) in testing them in only one. Perhaps that’d be worth a follow-up study, though I suspect it’d show the obvious: if you take intro phil, you’ll be better at thinking critically about intro phil, but if you take critical thinking, you’ll be better at thinking critically about critical thinking, with little to no significant difference in cross subject comparison.
Agreed.
In the surveys, students evaluated arguments that are familiar in everyday life. Here are 2 examples.
There was a large and significant difference in how the training and control groups evaluated these arguments 16 months after the semester was over.
Might any critical thinking class help students reason better about these issues? Yes. Of course.
But if Abstract Skeptic wants to claim that such issues are only relevant in my “fun slogan critical thinking” class, I’ll take that as a big win.
Our question was: Do the reasoning skills students learn in this class endure?
Your question is different and interesting: Does this critical thinking class (informed by Strategic Reliabilism) work better than other critical thinking classes (perhaps informed by other epistemological theories)? This study gives us no evidence on this question.
As you note, you could address your question by comparing my class to a class that uses (say) a popular critical thinking textbook. But you wouldn’t want *me* teaching the traditional class. You’d want someone who has experience teaching that class and who believes in it.
There are many critical thinking skills I don’t teach, including skills one might think of as more advanced or general. And I don’t teach epistemic virtues. I focus on teaching students skills students think they already know and use – and for most of these skills, they do! But they use these skills inconsistently. Sometimes very inconsistently. And so in class they practice, practice, practice using those skills on a wide range of cases. And they get pretty good at using those skills on the full range of cases to which they apply.
Sometimes, in working through cases, the students have to use more general or abstract skills, like trying to figure out: “What exactly is the issue / conclusion here?” And so I think it’s plausible that practicing the “easy” skills requires practicing (at least some) more general or advanced skills.
For me, the question you’re raising is: Does all this practice (and all this failure and feedback) indirectly promote improvements on other skills – including skills we might think of as more advanced? And does it (indirectly) promote more epistemic virtue? Honestly, I don’t know. (Though I have some suspicions.) But I hope so. Because I’m skeptical about my ability to teach epistemic virtue and these more advanced skills directly.
The point of our study was not to assess whether students in one philosophy class could do well on a test covering material from a different philosophy class.
The point of our study was to assess whether the skills students learned in a critical thinking class would endure long after the class was over. The point of having control students was to have a baseline of how students who hadn’t learned anything from the critical thinking class would do on the surveys.
I have read some of this literature and consistently find it to be very poor quality, with lots of non-sequiturs and very little engagement with the techniques actually used in domain-neutral critical thinking classes and the evidence that they are effective. On the former, see Martin Davies’ response to one of these educational psychologists here – https://philarchive.org/archive/DAVAQA-2. On the latter, argument diagramming is probably the most strongly supported domain-neutral technique (see for example this https://www.nature.com/articles/s41539-018-0038-5) yet is largely ignored by the anti-domain neutral crowd.
My intuitions are with Abstract Skeptic, but I’m open to be convinced otherwise. I suppose the reason my intuitions align with them is because I have all too often seen people confident in their ability as “critical thinkers” come to assess domain-specific information poorly, but with a level of confidence that suggests they believe that merely by being a “critical thinker” their analysis was worthwhile—oftentimes, it’s just shallow. But maybe that’s a separate issue altogether, having to do with other characteristics, rather than critical thinking as such.
I don’t see how the phenomenon you report is relevant to the domain-neutral vs. domain-specific debate. Does the converse phenomenon—people confident in their domain-specific ability making reasoning mistakes—undermine the domain-specific side? I don’t think so.
However, separate from this debate, the phenomenon you point to might be a reason to question the value of critical thinking courses. Likely what in going on in the cases you have observed is that someone has mastered a bunch of critical thinking skills but has not developed the intellectual virtues required to effectively put those skills to use and thus has acted in the intellectually arrogant way you describe. This problem is well known in critical thinking pedagogy and the solution is that a well-designed critical thinking course must not only develop critical thinking skills, it must also inculcate the intellectual virtues required to effectively put those skills to use. There is a significant amount of literature on how to do this, but unfortunately, not all critical thinking courses offered by philosophy departments are following best practice here, and some even ignore intellectual virtue entirely.
When thinking about domain-general vs domain-specific reasoning, I think it’s useful to start with a concrete example. An upshot of the opportunity cost principle, as I understand it, is: When you’re reflecting on an important decision, don’t just analyze a single option.
So when someone makes the case for or against an option after only analyzing the pros and/or cons of that option, 16 months after my class is over, my students find that reasoning far less persuasive than control students.
It’s a simple rule that can reasonably be applied to any decision you can reflect on and that’s worth reflecting on. Is this is a domain-general or domain-specific mode of reasoning?
To be clear, my point was only that this hypothesis made sense of the data presented by Bishop et al, not that it is true. Moreover, as a result, this study does not provide evidence that students improved in their general critical thinking abilities.
But I’m not an expert in educational psychology, so I can’t say whether the hypothesis is true. I guess some work by domain-specific authors is poor quality, but I think some is not, or at least, it is not by all appearances to this mere philosopher.
The argument diagramming paper you mention is interesting, and I agree the technique is notable. That said, consider the method used in their classes. Every session, they practiced applying the skill of argument diagramming to new examples, and they received detailed feedback on their efforts. As a result, they were able to apply that skill to a new setting (LSAT questions).
I think we should all agree that lots of practice and detailed feedback on applying a domain-neutral technique (logic?) to many different examples is a great way of acquiring at least something like the general critical thinking ability that’s marketed in our course objectives. After all, how did any of us get it?
I don’t see a hypothesis you’ve proposed that “made sense of the data” we report. Sometimes it seems like it’s:
H1. Our study didn’t establish that students improved on skills that we didn’t test for (or that weren’t taught in the class).
Fair enough. But that doesn’t make sense of our data. It just points to an open question raised by our study. (And, no, the skepticism check results don’t support H1.)
On occasion, you seem to be pushing a stronger hypothesis (to which Matthew is objecting). Something like:
H2. Our study hasn’t established that it’s possible to teach “general critical thinking abilities.”
This makes sense of our data only if none of the critical thinking skills we taught were “general.”
As I pointed out in my reply to Felix, the opportunity cost rule (which students were still using 16 months after the class was over) is extremely general in its application. This raises a dilemma: Count this critical thinking skill my students learned as “general” and H2 does not make sense of the data we report. Don’t count it as as “general,” and you owe us an explanation of what you mean by a “general” critical thinking ability.
These are great test questions. I might use them myself. One small quibble with the sunk cost question: The sunk cost fallacy applies well in financial contexts where money has already been spent (lost). But, for lives already lost in a war that is still ongoing, I think that the significance of the sacrifice of life can be enhanced some if the war is ultimately won, and I think that this is one factor that can reasonably weigh in favor of continuing a war. It might not be a heavy factor, and it can certainly be overridden by factors that weigh against continuing a war. But it should not be completely ignored for future decision-making in the same way that a non-refundable deposit that has already been paid should be completely ignored for future decision-making.
I know there’s a serious debate on the sunk cost principle in the philosophical literature. And I do mention this to my students (with links to some of the excellent papers on the topic).
It might be unrealistic to hope we won’t get into overly deep waters here, but here’s how I understand your example. The “enhancement” of the sacrifice is a benefit that’s conditional on winning the war. IOW, it’s a future (possible, likely, maybe even guaranteed) benefit. And so taking it into account does not violate the sunk cost principle.
Thanks, Michael. Yes, that is what I was thinking.
Someone help me with this sunk cost case
My confusion is that deposits are usually toward the total cost of the rental, aren’t they? So Sam will end up spending 500 extra on the whole vacation if he goes to the mountain vs the beach (assuming these otherwise cost the same). How this not a consideration in favor of the beach vacation? Surely budgeting, or trying to spend less money in total, is not an instance of the sunk cost fallacy?
I’ve never rented a vacation home so maybe I don’t understand how deposits work and/or how people are meant to reason about which vacation home to rent.
Overall I really like the teaching design and cases.
FWIW, one of the JEPA referees made exactly this point! You’re right that the non-refundable $500 deposit will make the future cost of the beach house $500 cheaper than it would have been had Sam not put down the deposit. The (lower) future cost of the beach house *is* relevant to Sam’s decision, and taking it to be relevant is not a violation of the sunk cost principle.
So assuming the original prices were identical, Sam now has to pay (say) $1000 for the beach house and $1500 for the mountain retreat. Those future costs are relevant to Sam’s decision. And by taking them into account, Sam is not violating the sunk cost principle.
The best way for philosophers to make sense of this (imo) is to distinguish between theoretical and practical reasoning. Sunk costs are 100% relevant to my (theoretical) reasoning about how much I still owe on my car, my house, or the beach house I’ve put a deposit on.
But once I’m clear about the future costs and benefits of my various options, sunk costs should not incline me (practical reasoning) to stick with my current car, my current house, or my current plan to go to vacation at the beach house.
In France there’s a very new project called “Cogito” that got 2 millions $ in crowdfunding to build a Critical Thinking curriculum to make available to schools. It’s in French unfortunately but their approach is very good imo.
https://www.kisskissbankbank.com/fr/projects/lespritcritique
https://www.cogito.fr/