The Dangers of Data on Teaching in Higher Education
“The dirtiest secret in higher education is that there is no good data on the quality of teaching and teachers on college campuses.”
So begins an interesting essay, “Teaching Quality,” by Hollis Robbins at her newsletter, Anecdotal Value.
Robbins laments the lack of data on instructional quality in higher education. Widely used student evaluations of teaching aren’t helpful, she says, owing to them “being designed to measure [student] satisfaction, not [teacher] quality.” Furthermore, their results are largely unavailable to those outside of the institutions in which they’re administered. What Robbins is looking for is data from which those inside and outside institutions can come to know how good the teachers are at them.
In the absence of such data, she estimates, based on her observations over 30 years in academia as a professor and dean, “that except for very elite private institutions… well over half of university instruction across the US is fair to poor. Perhaps 25% is good and 5% is excellent.” Your own estimates may vary.
Robbins takes the lack of data as a problem (for example, she asks who is to “blame” for it), but we might pause right here and ask why the lack of data is a problem.
It seems that Robbins takes the lack of data about the quality of college teaching to be related to both the lack of knowledge about the quality of college teaching and the persistence of the (in her estimation) low general quality of college teaching. Further, it is clear that Robbins thinks the existence of such data would be ultimately conducive to better teaching, and enable institutions of higher education to assure students, parents, lawmakers, and the public that college (suitably reformed) is worth it, even in an era of AI and corporate alternatives.
I think there’s some room for questions in this analysis.
“Pick Up Your Pencils, Begin” by Harriete Estelle Berman (installation & detail)
First, we might recall that data and knowledge are not synonyms, and that a lack of the former does not imply a lack of the latter. Data has its benefits, but what Robbins needs to get us to believe is that the marginal informational value of data, on top of what we already or could know without it, outweighs its costs (not just the costs of acquiring it, but especially the costs of the various ways in which, once it exists, it might be used—more on that later).
So what do we already know, and who knows it? Don’t at least some departments know which of their teachers are generally better or worse, and in which respects? Don’t they have means, such as classroom observations, that could be put to use in learning about teacher quality? Might students’ demonstrations of knowledge and skill in a course with a prerequisite be indicative, to some extent, of the quality of the teaching of that prerequisite? Which professors do students choose to take independent studies with? Hollis may rightly scoff at student evaluations of teaching—they do have their problems—but similar answers from multiple students to specific qualitative questions about teaching can indicate particular teaching problems (e.g., doesn’t provide feedback on work) or strengths (e.g., met regularly with students outside of class), even if such information isn’t presented in a wholly quantifiable form.
Aren’t these and other formal and informal means productive of some knowledge about teacher quality?
I assume that Robbins would say that the problem is that this knowledge is not readily exportable to those outside the institutions, and as a result, it is not readily usable as a means by which to hold the institutions accountable for the quality of their teaching. Data can be a tool for accountability.
It can be. But will it be? Is the kind of accountability such data facilitates likely to produce better teaching? Maybe, but I don’t see why we should assume it would.
Faculty are part of the problem, according to Robbins, since they “want even less attention paid to teaching quality than their institutions do.” Perhaps, but why? Part of the explanation, not mentioned by Robbins, must be the professional incentives faculty face. If universities started promoting people on the basis of their teaching rather than their research, more faculty would care more about teaching, and probably be better at it. Here the solution is a change in tenure and promotion policy, not the acquisition of more data.
But perhaps faculty are indeed resistant to more data collection about teaching. If they are, I think that is in part explained by quite reasonable concerns about how the data would be collected and used (concerns which are related to my earlier comments about costs and accountability).
The most significant concern is sometimes referred to as surrogation: the phenomenon of a measure of a quality replacing, in practice, the quality itself.
Surrogation contributes to what’s sometimes known as Campbell’s Law:
The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.
Any observer of K-12 education in the US is familiar with examples of surrogation and Campbell’s Law, as students at every level are subject to curricula aimed not at producing learning itself, but at producing high scores on standardized exams put in place to measure learning. Such exams were implemented to generate data in response to real problems with educational quality and fairness. But I don’t know anyone who would look at the “teaching to the test” such well-meaning data collection has wrought and be happy about it. Do we think our young students are learning better under a data-dominated educational regime? Do we think our teachers are teaching better under it? No and no.
The idea of a large scale attempt to gather data about teaching quality in higher education raises similar prospects. What are the instruments by which such data would likely be collected? Standardized exams? Why wouldn’t this reproduce the same counterproductive and unintended consequences it has at lower educational levels? Shouldn’t that prospect have us also worried about reduced academic freedom in teaching, and about the erosion of a valuable pluralism within and across institutions?
I understand that Robbins sees data as a key to the survival of higher education, as a way of showing the rest of our culture that going to university is still worth it. But to me it just further adds to the likelihood that higher education ends up drowned after suffering too many hits from carelessly tossed lifesavers (like “tell them to go to college to get a good job!” or “become an ‘AI-powered’ university!”).
Because Robbins has a dim view of current teacher quality, she says:
Lawmakers should focus less on politics and efficiency and start asking for value-added learning data. Parents should start asking, “How do you guarantee the quality of the professors my college student will face?”
I don’t disagree that teaching and learning at the university level could improve. But it seems unlikely to me that the blunt data-collecting instruments most likely to be demanded by politicians and most likely to be marketed to parents as a simple way to choose among colleges will lead to such improvement. Once the “value-added” by a university education is surrogated by data gleaned from standardized exams, universities become even more vulnerable to replacement by more “efficient” means to higher test scores.
If Robbins hopes to save universities a space in a world that increasingly treats higher education as vocational training and that increasingly invites AIs to do our thinking and communicating for us, calling for a metric that will reduce all the various things a university can do for students to a few numbers does not seem like a promising strategy.
But it could be that I’m overly pessimistic or unimaginative in considering the options. Discussion welcome.
I appreciate your pushback Justin and I ought not to assume that all readers understand that I’ve been publicly critiquing data collection in my writing (Compact, CHE) for over a year.
My point is that not having a conversation about teaching quality harms faculty. I am seeing across the country policies to cancel classes with fewer than 15 students, to increase lecturer teaching loads without increasing pay, without anyone asking “how can there be quality teaching under these circumstances?” I am seeing institutions optimized for completion, not education. Again, if the goal is completion, why should faculty give it their all, as if the goal were education?
The best teaching happens with engaged students who appreciate faculty giving it their all. My goal was starting a conversation, not necessarily suggesting more metrics, so I very much appreciate your post.
Thanks for the comment, Hollis, and for the links to a couple of your other pieces. In your Compact piece, you write:
I agree with the general thrust of this point. My concern about your call for data is that, in virtue of its portability and apparent scientific imprimatur, data is a tool for centralization. You seem to acknowledge this and other worries about data in the Compact piece.
So perhaps I interpreted your references to “data” too narrowly. In that case, though, I don’t quite know what you mean when you call for “good data” or for lawmakers to ask for “value-added learning data”.
(By the way, I do not mean to convey by my post that I’m in some general way “anti-data.” I’m not. My point is only that, as with many things, even with the gathering and sharing of information, there are tradeoffs that need to be taken into account.)
All good I packed a lot of points in, and part of my critique is against those since Buckley, as I say, who confuse teaching quality with course content. They don’t even try to find data to support the idea that teaching more Burke and Oakeshott is “better” teaching. I may not disagree but there is zero data.
Thanks for posting on this really interesting essay–and thanks for the essay to begin with!
I find a lot to agree with, in both the essay and the post. For example, I agree that: teaching quality is uneven; that it is important; that it is difficult to quantify and track; that data can distort; and also that data can be helpful, especially in communicating value to students, parents, politicians; that institutions aren’t transparent about teaching quality; and that there are real and persistent structural barriers to improving teaching quality.
I do, however, want to push back a bit on some of the claims (and assumptions) in both the post and essay. Much of what I say is based on my work as director of the teaching and learning center at my own institution, which includes both assessment of what’s happening on my own campus and familiarity with some of the relevant literature.
First, I think it’s extremely important to keep in mind that elite institutions educate about 1% of college students in the US. They also, because they are elite, educate a different student body. This has implications for claims about what quality teaching looks like, how much it varies, and also how to measure it.
(As an aside and in defense of my talented colleagues: I would need to be given some good reason to agree with Robbins’ assertion that teaching quality is higher at elite institutions than at all the rest. Why, aside from your own perceptions given you “perspective,” say such a thing?)
For example, I submit that the oft-cited Carrell and West study Robbins cites in her essay is not actually as apt to support the general conclusions the authors and many others take it to. USAFA students are, as detailed in that paper itself, high achievers that are nothing other than geographically representative. They skew elite, with all of the demographic homogeneity and other characteristics that come with that. This is an important limitation of that study and should be kept in mind when thinking about generalizing the findings. Furthermore, given what we know about who tends to teach intro courses and who teaches successor courses, including at USAFA as discussed in that paper, it is possible that performance in an intro course taught by an adjunct prof with no terminal degree may not predict performance in successor course taught by a research-focused prof with a terminal degree, and this may have nothing to do with the first prof’s quality as a teacher. The two may have different teaching styles, expectations, etc, and these factors may influence performance (eg, one learns how to study for exams given one type of preparation but this doesn’t translate to another type of prep), let alone SET ratings. This issue is exacerbated when some of the students in the successor course are having this prof for a second time, while others are meeting them and their methods for the
first time. In short, that study has some elements of good design but also
important limitations and potential confounds, some of which are due to the
type of institution it looked at. Proceed with caution.
In general, we should treat studies focused on elite institutions with extreme caution. When it comes to data about student learning, teaching quality, and so on, elite schools are just not at all representative samples. To illustrate further: one of the tenets of the anti-SET crowd is that SET ratings correlate with expected grades, and this is taken to be evidence that SETs contribute to grade inflation. Well, what is someone at a school where the students expect, based on their high school and college experiences, to receive a wide range of grades supposed to learn from a
study that looks at a school where the students all get nothing but As in high school and college and so expect this to continue? The grade expectations on the part of students at these different types of institution just aren’t the same. This confounds our ability to apply conclusions about correlations in one context to the other.
This leads me to a second point. I think folks are way too quick to dismiss student ratings as sources of data about teacher quality. (This is to agree with some of what Justin says in his post.) Sure, students may give most instructors pretty high ratings. But we can track differences between pretty-highly-rated instructors. And we can invite students (and our colleagues) to help us do so. This need not be a simple matter of quantitative scores on end-of-term surveys. It may also involve things like asking our students open-ended questions and being open to their unvarnished responses. When my colleague and I sent out a survey asking students about their decisions to drop or remain enrolled in courses, we got a lot of good info. Teaching quality was amongst the most important factors. And their responses to open-ended prompts revealed lots of great details about what quality teaching looked like to them.
On this same point, it is striking that many of the studies looking at SETs, which form the basis for dismissing them on the grounds that they are biased and feed grade inflation, show statistically significant correlations between things like expected grades or student/instructor gender and SET ratings. But very few contextualize the variance. The much-discussed recent report on grade inflation out of Harvard, for instance, contains a note at the end that shows that grades are significant predictors of SET scores but that they only predict 1% of variance. 1%! I fail to see how this is supposed to show, as many have taken it to, that SETs feed grade inflation. Perhaps they do so indirectly, via faculty’s perception that there’s a quid pro quo here. But that perception seems, given the data, to be based on misunderstanding.
Third point: what our students said when we asked them about dropping courses tracked with the vast literature on teaching effectiveness. I really don’t know why the canard that we don’t know what effective teaching is still sucks up so much oxygen. Is there complete agreement? No. But there’s never complete agreement about anything in academia. That doesn’t mean there isn’t lots of agreement or even data to go on. Robbins cites some stuff, such as the importance of active learning techniques. But there’s lots of other stuff, too. Look at books by Bain (2004), Lang (2021), hooks (1994), Bonwell and Eison (1991), etc. Read some SoTL journals. Attend some sessions at a Lilly conference. Observe a class taught by a colleague who is well-liked by students. Think about your own experience and replicate what worked when you were a student. Most of these
sources of info about quality teaching track basically the same dispositions
and practices, and there is (despite what others say) loads of data that they work in terms of metrics like performance on assessments, success in subsequent courses, etc.
I want to (finally!) wrap up by echoing a deeply important truth in Robbins’ essay: teaching quality matters, not just for student learning, but also for the survival of higher ed. I was recently in a meeting where a brave person said the quiet part out loud: “Do we really even believe in the product anymore?” This was a conversation about student enrollment and retention. And though I bristle at the commercial framing here, I wholeheartedly agree with the underlying sentiment that it’s difficult to stand up for something you don’t believe in yourself. Research is important. I’m not advocating for giving up on that. But without students, there are no universities. And without quality teaching, there will be fewer and fewer students. While Justin is right that data can distort, and it is crucial that we take care not to let data on teaching quality be weaponized by those who have it out for higher ed, it is also crucial that we heed what defenders of tenure have been saying for decades: the best way to protect academic freedom and academia more broadly is to hold ourselves to high standards. This goes for teaching just as much as, if not more than, anything else we get paid to do.
I remember a study showing that research quality has an overall positive correlation with teaching quality, measured by students’ competence in follow-up course, but a slightly negative correlation with students’ satisfaction. This, I guess, is one of many studies suggesting unreliability of students evaluation, though nobody thinks evaluation has no value (if some do, they are overly loud; the vast majority of us do take it seriously for what it is).
“The idea of a large scale attempt to gather data about teaching quality in higher education raises similar prospects. What are the instruments by which such data would likely be collected? Standardized exams?”
This is basically what the University of Oxford does. (Teaching decentralized to dozens of people across the colleges; exams set centrally.) I would defend the claim that its teaching is pretty high quality.
Oxford is not representative of institutions in the US (it is not even representative of institutions within the UK), and standardisation across one institution is not at all like standardisation across a nation of several thousand institutions of higher education. So it is unclear whether this example can be generalised to assessing teaching quality across the US.
“Don’t they have means, such as classroom observations, that could be put to use in learning about teacher quality?”
If the classroom observations are to provide fair and accurate assessments, they may require a large time commitment by observing faculty, who ought to attend multiple class sessions and review teaching materials. Attending one or two classes, reviewing a syllabus and checking grade distributions for one course won’t suffice. Will allowances be made for that effort, which will reduce time available for other professional responsibilities?
“… similar answers from multiple students to specific qualitative questions about teaching can indicate particular teaching problems ….” They can indeed, and published research concerning a variety of student surveys indicates that there is good reason to be very cautious because of the prevalence of various kinds of biases and other memory issues. Some of the relevant citations:
In 2007, Stephen R. Porter began to raise serious questions about the trustworthiness of factual information reported in student survey results. [http://chronicle.com/article/Education-Researchers-Group/129296/ and https://www.insidehighered.com/news/2011/10/07/assuring-civility-or-curbing-criticism ]
Stephen R. Porter, “Do College Student Surveys Have Any Validity?” The Review of Higher Education v5 n1 (Fall 2011) 45–76
“In this article, I argue that the typical college student survey question has minimal validity and that our field requires an ambitious research program to reestablish the foundation of quantitative research on students. Our surveys lack validity because (a) they assume that college students can easily report information about their behaviors and attitudes, when the standard model of human cognition and survey response clearly suggests they cannot, (b) existing research using college students suggests they have problems correctly answering even simple questions about factual information, and (c) much of the evidence that higher education scholars cite as evidence of validity and reliability actually demonstrates the opposite. I choose the National Survey of Student Engagement (NSSE) for my critical examination of college student survey validity ….”
{Cited in an early, but not the final, version by Porter:
Garry, M., Sharman, S. J., Feldman, J., Marlatt, G. A., & Loftus, E. F. (2002). “Examining memory for heterosexual college students’ sexual experiences using an electronic mail diary.” Health Psychology, 21(6), 629-634. Abstract: To examine memory for sexual experiences, the authors asked 37 sexually active, nonmonogamous, heterosexual college students to complete an e-mail diary every day for 1 month. The diary contained questions about their sexual behaviors. Six to 12 months later, they returned for a surprise memory test, which contained questions about their sexual experiences from the diary phase .… [Except for their accurate recollection of (low) frequency of anal sex, the students grossly overestimated by as much as a factor of four the frequency of vaginal or oral sex; men and women did not differ significantly in their overestimates.]}
Stephen R. Porter, Corey Rumann, and Jason Pontius, “The Validity of Student Engagement Survey Questions: Can We Accurately Measure Academic Challenge?” New Directions for Institutional Research, n150 (Summer 2011) 87- 98 DOI: 10.1002/ir.391
This chapter examines the validity of several questions about academic challenge taken from the National Survey of Student Engagement. We compare student self-reports about the number of books assigned to the same number derived from course syllabi, finding little relationship between the two measures.
Stephen R. Porter, “Self-Reported Learning Gains: A Theory and Test of College Student Survey Response,” Research in Higher Education (Nov 2012) DOI 10.1007/s11162-012-9277-0
Abstract: Recent studies have asserted that self-reported learning gains (SRLG) are valid measures of learning, because gains in specific content areas vary across academic disciplines as theoretically predicted. In contrast, other studies find no relationship between actual and self-reported gains in learning, calling into question the validity of SRLG. I reconcile these two divergent sets of literature by proposing a theory of college student survey response that relies on the belief-sampling model of attitude formation. This theoretical approach demonstrates how students can easily construct answers to SRLG questions that will result in theoretically consistent differences in gains across academic majors, while at the same time lacking the cognitive ability to accurately report their actual learning gains. Four predictions from the theory are tested, using data from the 2006–2009 Wabash National Study. Contrary to previous research, I find little evidence as to the construct and criterion validity of SRLG questions.
Porter’s conclusions about student surveys receive strong support in this doctoral dissertation:
William R. Standish, III, A Validation Study of Self-Reported Behavior: Can College Student Self-Reports of Behavior Be Accepted as Being Self-Evident? (2017)
https://repository.lib.ncsu.edu/bitstream/handle/1840.20/33607/etd.pdf?sequence=1
Abstract excerpt: This validation study of self-reported behaviors compares institution-reported, transactional data to corresponding self-reported academic performance, class attendance, and co-curricular participation from a sample of 6,000 students, using the Model of the Response Process by Tourangeau (1984, 1987). Response bias, observed as measurement error, is significant in 11 of the 13 questions asked and evaluated in this study. Socially desirable behaviors include campus recreation facility (CRF) use and academic success being overstated as much as three times. Nonresponse bias, observed as nonresponse error, is also significant in 11 of the same 13 questions asked and evaluated with high GPA and participatory students over represented in the survey statistic. For most of the questions, measurement error and nonresponse error combine to misstate behavior by at least 20%. The behaviors most affected are CRF use, which is overstated by 112% to 248%; semester GPA self-reports of 3.36 versus an actual value of 3.04; and co-curricular participation that misstated by between -21% to +46%. This validation study sufficiently demonstrates that measurement error and nonresponse error are present in the self-reported data collected for the commonly studied topics in higher education that were represented by the 13 questions. Researchers using self-reported data cannot presume the survey statistic to be an unbiased estimate of actual behavior that it is generalizable to larger populations.
[Porter was a committee member, but not the dissertation director. Before beginning his doctoral dissertation research, Dr. Standish was an experienced higher education data analyst.]
Porter is not the only researcher who has highlighted problems with student surveys as information sources:
NA Bowman, “Can 1st-Year College Students Accurately Report Their Learning and Development?,” American Educational Research Journal, v47 n2 (2010) 466-496.
Abstract: Many higher education studies use self-reported gains as indicators of college student learning and development. However, the evidence regarding the validity of these indicators is quite mixed. It is proposed that the temporal nature of the assessment—whether students are asked to report their current attributes or how their attributes have changed over time—best accounts for students’ (in)ability to make accurate judgments. Using a longitudinal sample of over 3,000 first-year college students, this study compares self-reported gains and longitudinal gains that are measured either objectively or subjectively. Across several cognitive and noncognitive outcomes, the correlations between self-reported and longitudinal gains are small or virtually zero, and regression analyses using these two forms of assessment yield divergent results.
See also:
NA Bowman & TE Seifert, “Can College Students Accurately Assess What Affects Their Learning and Development?” Journal of College Student Development, v52 n3 (May-Jun 2011) 270-290; and NA Bowman, “Examining Systematic Errors in Predictors of College Student Self- Reported Gains,” New Directions for Institutional Research n150 (Sum 2011).
Shana K. Carpenter, Amber E. Witherby and Sarah K. Tauber, “On Students’ (Mis)judgments of Learning and Teaching Effectiveness,” Journal of Applied Research in Memory and Cognition 9 (2020) 137–151.
Abstract: Students’ judgments of their own learning are often misled by intuitive yet false ideas about how people learn. In educational settings, learning experiences that minimize effort and increase the appearance of fluency, engagement, and enthusiasm often inflates students’ estimates of their own learning, but do not always enhance their actual learning. We review the research on these “illusions of learning,” how they can mislead students’ evaluations of the effectiveness of their instructors, and how students’ evaluations of teaching effectiveness can be biased by factors unrelated to teaching. We argue that the heavy reliance on student evaluations of teaching in decisions about faculty hiring and promotion might encourage teaching practices that boost students’ subjective ratings of teaching effectiveness, but do not enhance —and may even undermine — students’ learning and their development of metacognitive skills
General Audience Summary: As the changing landscape of education provides more freedom and flexibility in the options available to students, it is becoming increasingly important that students be able to successfully evaluate and manage their own learning. This is easier said than done, however, because students often misjudge their own learning of a given topic to be better than it actually is. This common tendency toward overconfidence can be further bolstered by a number of intuitive but misleading factors that enhance students’ subjective impressions of how much they have learned, without always enhancing their actual learning. Students believe, for example, that they learn best from enthusiastic and engaging instructors who provide smooth and well-polished lectures that do not require active class participation. Such factors, although they readily inflate students’ judgments of their own learning, do not consistently enhance students’ actual learning. They also inflate students’ evaluations of the effectiveness of their instructors. Indeed, students’ evaluations of teaching effectiveness can be poor predictors of their actual learning in their courses, and these evaluations can be biased by external factors unrelated to student learning, such as an instructor’s gender, age, attractiveness, and grading leniency. Given the heavy reliance on student evaluations of teaching effectiveness in decisions regarding faculty hiring and promotion, faculty may be incentivized to adopt teaching approaches that boost their evaluations but do not enhance — and could even undermine — students’ academic success.
Despite the relatively small size of the populations involved, both of the following studies highlight disturbing possibilities:
M. Oliver-Hoyo “Two Groups in the Same Class: Different Grades.” Journal of College Science Teaching, 38(1) (2008) 37-39. [-recounts the experience of one award-winning instructor who taught two sections in the same classroom at the same time and received significantly different evaluations from the two sections. One of the matters about which there was disagreement was how available the instructor was outside of class. All students were informed in the same way about the instructor’s office hours, she kept them, and many students from each section made use of them. There was a significant association between receiving lower grades and rating the instructor as significantly less available during office hours.] http://www.ncsu.edu/chemistry/people/moh.html
Robert J. Youmans and Benjamin D. Jee, “Fudging the Numbers: Distributing Chocolate Influences Student Evaluations of an Undergraduate Course,” Teaching of Psychology v34 n4 (2007) 245-247.
Abstract: Student evaluations provide important information about teaching effectiveness. Research has shown that student evaluations can be mediated by unintended aspects of a course. In this study, we examined whether an event unrelated to a course would increase student evaluations. Six discussion sections completed course evaluations administered by an independent experimenter. The experimenter offered chocolate to 3 sections [immediately] before they completed the evaluations. Overall, students offered chocolate gave more positive evaluations than students not offered chocolate. This result highlights the need to standardize evaluation procedures to control for the influence of external factors on student evaluations.
See also: Michael Hessler et al, “Availability of [chocolate] cookies during an academic course session affects evaluation of teaching,” Medical Education (2018) doi: 10.1111/medu.13627 9 pp.
Concerning student evaluations of teaching, it is also worth noting that the results can easily be unfair even if the evaluations are unbiased, and relatively reliable and valid:
Justin Esarey and Natalie Valdes, “Unbiased, Reliable, and Valid Student Evaluations Can Still Be Unfair ,” Assessment & Evaluation in Higher Education (Published online: 20 Feb 2020) DOI: 10.1080/02602938.2020.1724875
Abstract: Scholarly debate about student evaluations of teaching (SETs) often focuses on whether SETs are valid, reliable and unbiased. In this article, we assume the most optimistic conditions for SETs that are supported by the empirical literature. Specifically, we assume that SETs are moderately correlated with teaching quality (student learning and instructional best practices), highly reliable, and do not systematically discriminate on any instructionally irrelevant basis. We use computational simulation to show that, under ideal circumstances, even careful and judicious use of SETs to assess faculty can produce an unacceptably high error rate: (a) a large difference in SET scores fails to reliably identify the best teacher in a pairwise comparison, and (b) more than a quarter of faculty with evaluations at or below the 20th percentile are above the median in instructional quality. These problems are attributable to imprecision in the relationship between SETs and instructor quality that exists even when they are moderately correlated. Our simulation indicates that evaluating instruction using multiple imperfect measures, including but not limited to SETs, can produce a fairer and more useful result compared to using SETs alone.
[Despite having been on the faculty of the same institution as Porter and Oliver-Hoyo, I have never met either of them in person. I’ve corresponded briefly with Porter and Bowman. I met Dr. Standish once during an in-person committee meeting.]
I agree that surrogation is a problem, but we don’t seem to take that worry to be decisive when it comes to the evaluation of our students. We often evaluate students based on tests or assignments that are surrogates for the thing itself. These evaluations become data about the achievements of the student, and many students try to game the system in educationally unproductive ways. But we haven’t given up on evaluating students because we feel that whatever the evils created by the particular surrogate we pick, it’s the best option we have.
Why shouldn’t we say the same thing about evaluation of teachers? There are some bad surrogates we shouldn’t use. There are some good surrogates that are too costly. Why not choose some happy-medium surrogates and keep thinking about how to improve them?
The biggest obstacle I can see in philosophy is that we have much less agreement on what a good education should look like than in some other disciplines. But the level of disagreement in philosophy has never stopped us from thinking that we can evaluate our students. Why shouldn’t we apply the same approach to teaching?
I say this as someone who is skeptical of outcome-based evaluations of student learning in philosophy. I’m very much opposed to applying anything like K-12 standardized testing to students as a measure of teaching. I think classroom observations are probably essential to any data worth generating, and that is costly. But I see an urgent need for some sort of teacher-evaluation system because higher education has become extremely costly, and its cheap rivals are becoming increasingly effective.
A few elite institutions have no need for this. They can market themselves on the quality of the faculty’s research: Come study here not because we try to make sure our faculty teach well, but because you want to be at the cutting edge. Other institutions can market themselves based on their parties or their sports. But many students (and their parents) aren’t primarily interested in research, parties, and sports. They’d rather have good teachers.