What I sometimes call the “great Millian hope” is that freedom, knowledge, and happiness are positively correlated. A new study conducted by a philosopher and a psychologist provide some reasons to think that, in the college classroom at least, the hope can be realized.
–
In “Choosing to learn: The importance of student autonomy in higher education“, published yesterday at Science Advances, Simon Cullen and Daniel Oppenheimer (both at Carnegie Mellon University) report on a pair of studies they conducted about college student autonomy. In one, the students were given the choice of whether to have their particular class attendance mandatory and factored into their overall course grade. In the other, they gave students the option to opt out of some more challenging work.
From the abstract of their paper:
We demonstrate how two autonomy-supportive policies effectively increase classroom attendance and subject mastery. First, in a randomized controlled field study, we explored the effect of allowing students to choose whether to make their attendance mandatory (i.e., a component of their course grades). We found that nearly all students used the opportunity as a pre-commitment device and were subsequently more likely to attend class than were students whose attendance had been mandated. Second, in a multi-year cohort study, we explored the effect of allowing students to opt out of a challenging, high-effort assessment stream, finding that students given greater autonomy invested more effort into their assignments and attained greater proficiency with the material.
And from the paper itself:
Allowing students to commit to mandatory attendance and to choose more rigorous assignments led them to attend class more reliably, to put more effort into their assignments, and to understand the material better.
Further, most students who chose mandatory attendance were happy with their choice (10% were not).
You should check out the full paper for details on their studies and findings.
Cullen and Oppenheimer think that the benefits of student autonomy probably apply beyond the domains in which they tested it. They write:
While we have focused on attendance and assessment, similar choice architectures can be applied to many other course elements. For any mandatory course element, it is worth considering whether making it optional-mandatory might serve students better.
They acknowledge that faculty may be reluctant to give students more freedom:
Faulty intuitions about the drivers of human motivation might also make faculty reluctant to grant students greater autonomy. For example, although people believe that they are more likely to persist in challenging behaviors such as dieting and exercise when they focus on long-term benefits, research suggests that attending to immediate rewards leads to greater persistence. Faculty might similarly turn to autonomy-undermining extrinsic motivators if they forget that learning should be inherently fun and rewarding. Moreover, research on preferences for paternalistic policies in educational contexts reveals that a meaningful fraction of people believe it is a professor’s job to protect students from making poor decisions in and even outside of the classroom. Thus, paternalistic attitudes could lead faculty to believe that they must make wise decisions on students’ behalf.
Of course, another possibility is that faculty want to give students more autonomy but lack tools for doing so.
To help on that last point, they offer some other examples of opportunities for increasing student choice:
| Intervention | Description |
|---|---|
| Deadlines | Instead of purely faculty-set deadlines, students could have a say in choosing their own, allowing for greater autonomy and better schedule management. |
| Extensions | Students could be given a number of “extension days” to use at their discretion over the semester, removing subjectivity from the extension process while boosting student motivation. |
| Materials | Rather than dictating course materials, instructors could offer multiple options and formats for readings and provide supplemental resources for students who want to deepen their understanding. |
| Exams and projects | Giving students a selection of exam questions or project formats can also enhance autonomy. Students could select which questions to answer or choose their final project medium, subject to instructor approval. |
| Assessment weights | Permitting students to allocate weight to different course elements within set bounds could enhance their sense of autonomy and motivation. |
| Supplementary topics | While core topics are nonnegotiable, supplementary topics could be chosen by student vote, increasing engagement by aligning course material with student interest. |
The full paper is here.
Discussion welcome, especially descriptions of how your own attempts to increase student autonomy have gone.
Lots to love about this idea. So, thanks for sharing!
I’m happy to report that I design lots of space for student autonomy in my classes. I allow students to choose materials we read/cover. I consult with them on exam design. I “task” them with designing and completing a collective final exam. I always negotiate deadlines. Class structure is constantly evolving, as I make it clear to the students class is “ours” and they should also feel encourage to suggest changes or comment on what works and what does not. All my classes are student-driven, which means I never lecture and employ lots of pedagogical improvisational to respond to where and how students go. Lots more, but you get the idea. As Rancière/Jacotot says, believe in their intelligence and don’t rob them of the opportunity to think and decide.
In my now ten years of teaching this way, the secret is to strike the right balance between just enough autonomy (which is never individual but always a shared/collective) so they can see themselves in the result but not too much so that they feel lost or overwhelmed.
This sounds awesome! I also elicit lots of feedback from students. I think it’s doubly useful: I (often) get to improve my approach, and my students can see their own influence how the course runs. The last lecture of the semester is always the “Student Choice Lecture,” where the class votes on the topic of the final lecture (an idea I got from Danny). It’s often my favorite lecture of the semester and usually makes its way into the regular rotation of the next iteration!
Have you written about your approach anywhere? I’d love to learn more.
One plausible principle of fair grading is that all students should be evaluated according to the same standards over the course of the semester. (I recall a lengthy defense of this idea in Daryl Close’s “Fair Grades” (2009).) Some of the authors’ proposals will foreseeably result in that not happening (except in the extremely rare case that all students pick the same option when given the choice). This means that end of semester grades will become less consistent across students since different grading standards will apply to students in the same class. Allowing students to choose how grades are weighted or whether certain parameters count at all may also lead to them choosing weighting schemes that arbitrarily inflate (or, if they exercise poor judgment, deflate) their grades. Maybe the benefits of promoting student autonomy are assumed to outweigh these drawbacks, or maybe the authors are not concerned about these aspects of impartial and consistent grading. In any case, the authors don’t seem to address this concern in the study.
I’ll also just note that I’d be hesitant to generalize the authors’ findings from a (representative?) sample of Carnegie-Melon students to undergraduate institutions more broadly. Given the recent problems with replication in the social sciences, data from other educational contexts is needed.
I don’t find much of the Close paper compelling, but in any case it might be worth noting that Close also takes the relevant principles to rule out grading attendance at all, so it’s a bit of a moot point with respect to the attendance stuff.
(Also, consider: I invite people to my wedding. On the invitation they can indicate what meal they would like: the pasta or the soup. Some choose pasta, others choose soup. Can I be charged with failing to treat my guests fairly and impartially, since they don’t all end up with the same dinner?)
Close’s approach rules out grading students exclusively on the basis of them being in class, but it does not rule out grading participation-related in-class activities, which is one method people often use to encourage attendance. The basic concern would still apply if students could choose whether (and to what extent) these activities counted toward their final grade.
I don’t follow the wedding analogy. You are not assessing your wedding guests’ mastery of anything. Fairness of the kind we’re talking about is not relevant to that case.
If you are allowing students to choose what assessments count toward their final grade (or how much those things count for), then students with similar levels of mastery could wind up with significantly different course grades. That’s the sense in which these grading schemes are “unfair” in this context. If grades are supposed to be an indicator of how much students know about the course content or the extent to which they have developed course-related skills, then this is a problem. Obviously, not everyone holds that view of grading, but I think that is at least a very large part of what I am doing when I assign grades to my students at the end of the term.
In the article, we suggested allowing students to choose how different course elements are weighted “within set bounds”. Whether this is appropriate will, of course, depend on the details of the class. But just to give a more concrete idea of what we were thinking: Suppose you have a class with (a) weekly short take-home assessments, (b) a longer midterm essay, and (c) a final exam.
You might decide that, for all students, the final exam will count for 20% of their final grade while allowing students to vary the weight of the take-home assignments (say, between 20–40%) and the midterm essay (say, between 60–40%).
I can see reasons for thinking this approach might increase fairness:
This kind of bounded flexibility doesn’t seem to entail lowering standards or creating unfair advantages. Instead, it might give all students the opportunity to put their best foot forward while maintaining the integrity and consistency of the assessment process.
I think this system might actually be fairer than a one-size-fits-all approach that can advantage some students because of circumstances unrelated to their mastery of the course material.
To add to your example, suppose two students both score 80% on the midterm essay and 90% on the short weekly assignments. That comes to a 2 point difference in their final grade if one weights the midterm essay at 40% and the other at 60%.
If they both score 100% on the final exam, the final grade of the student who chose to weight their midterm essay at 60% would be an 86 (B) while the final grade of the student who chose to weight their midterm essay at 40% would be an 88 (B+).
How is it not unfair to give two different final grades to students who scored identically on all assessments?
Why not think this approach is evaluating students on things unrelated to their mastery of the course material like their ability to predict their own performance and tactically choose assignment weights?
I don’t think it’s meaningful to say these students scored identically on all assessments, since in an important sense, these students had *different* assessments. If one student had a midterm essay weighted at 60% and the other had one weighted at 40%, then these are importantly different assignments that legitimately affect how much effort they put into them.
What? Two assignments identical in every respect except their ‘weight’ are the same assignment.
1. They’re not different assessments just because they’re weighted differently. I don’t have an ontology of assessments, so I can’t tell you exactly what marks the boundary between one assessment and another, but that can’t be it.
2. If the only difference between their performance on the midterm essay is how much effort they put into it (the 60% weighted student tried harder but produced the exact same quality essay as the 40% weighted student), that doesn’t justify giving them different final grades because “effort” is not an aspect of the course material.
A student who’s a philosophical genius and so doesn’t put any effort into their A quality essays still deserves an A. A student who works incredibly hard but is unable to produce work above a C level still deserves a C.
maybe this is not what is most important. what is most important is that students grow and develop intellectually. Grades are,nowadays at least, a hinderance to this as they encourage a view of college education as “work’ in the bad Marxian alienating sense .
Close’s excellent and influential essay takes a narrow, roughly Kantian approach to grading (as strictly desert-based) that many folks find overly restrictive and insensitive to context. Alternatives (e.g. approaches that use grades for motivation) can be effective though they are liable to introduce bias in various ways. Some of us believe that we are teaching people through philosophical content, rather than just teaching philosophical content. There are deep and fecund questions here, some of which are treated in a symposium on Close’s article in: Philosophy Through Teaching, edited by Esch, Hermberg, and Kraft.
Briefly: the truism that the same work should always receive the same grade ignores facts about: different student backgrounds, different educational contexts, different instructor expectations, and, sure, different student choices.
This is great – thanks, Justin! In a recent article published in the APA Studies on Teaching Philosophy, my colleague Karen Adkins and I describe an autonomy-focused intervention concerning student discussion assignments. Our findings over multiple semesters of using this intervention suggest that when instructors demonstrate that they trust students, the students are more invested in completing the work and (maybe more importantly for us) are invested in developing philosophical community beyond the classroom.
Thanks for sharing this article—I hadn’t seen this until now! I’m curious how students feel about the peer grading? It’s an interesting solution to free-riding.
Thanks for the question, Simon! Students do tend to appreciate the opportunity to evaluate their peers. They are expected to offer narrative feedback to support their quantitative evaluation, and those give good insight into what students come to expect from the assignment.
Anecdotally and over the last six years I’ve observed these peer evaluations to cluster in revealing ways. That is, for groups that everyone seems to be participating in regular ways the evaluations are all positive for each group member; for groups where a free rider is present, the students in that group offer evaluations of that student that reflect their recognition that there is a free rider in their group. This is helpful because the intuitive sense I have reading the group’s output over the course of the semester tends to match the way the evaluations shake out. This is a strong tendency and has been confirmed in all but a very few (memorable, frustrating!) cases.
Does increasing the number of micro-choices that students have to make increase their autonomy or displace it? I do wonder. Excited to read the paper.
This is beautifully put: “What I sometimes call the “great Millian hope” is that freedom, knowledge, and happiness are positively correlated.”
But you’ll forgive me for upping the ante just so: freedom, reason, and right.
Some context worth emphasising before everyone starts updating in favour of more autonomy = more learning in general:
a) It’s the autonomy being used for pre-commitment, not autonomy simpliciter, that does the work:
“making attendance fully optional preserves student autonomy but results in lower turnout “
[from another well-known paper] “The results from Study 2 show that performance under self-imposed deadlines is lower than performance under evenly spaced deadlines, but higher than performance under maximally delayed deadlines” https://doi.org/10.1111/1467-9280.0
b) Selection bias means that things that work for high-achieving students frequently do not work for lower-achieving students, who often benefit the most from structure and external motivation:
“Carnegie Mellon is a selective university… with nearly one in six students coming from outside the United States”
c) Timely reminder to always look at the results section yourself:
“Compared to students in the mandatory groups, optional-mandatory students were more likely to attend weekly sections [odds ratio (OR) = 1.7, 95% CI 0.96 to 2.9], but this effect did not meet the conventional standard of statistical significance: P = 0.072.”
a) As we mention in the results and discussion sections:
“It is also important to note that our findings cannot be entirely explained by the effectiveness of pre-commitment. In Study 2, students in the free-to-switch cohort showed greater engagement and improvement than those in the mandatory cohort, despite having the option to abandon their initial choice at any time before the midterm essay deadline. This suggests that autonomy itself, even in the absence of strong pre-commitment, can be a powerful motivator.”
What we’re advocating isn’t “anything goes” but using choice architecture to design policies where students are likely to make wise decisions for themselves. We can’t be sure, but we speculate about which features of the choice situation are responsible for the attendance effects:
“In a purely optional regime, students are forced to make difficult trade-offs repeatedly over the course of the semester (“Do I trudge through the snow to get to class or stay home in comfort?”). The more often students face such decisions, the more likely they are to sometimes choose not to come to class. By having students pre-commit to attendance when these trade-offs are not as salient, the choice architecture provides choice in a context when students are most likely to make educationally beneficial decisions. Of course, this is speculative, and future research should explore such mechanisms in greater detail.”
b) I’m happy to report that we’ve partnered with six institutions (so far!) – including community colleges – and preliminary data indicate that the uptake of optional-mandatory attendance is similar to what we’ve observed many times at CMU. Of course, as we outline in the discussion section, much more work is required to establish the generalizability of our results. But decades of research in self-determination theory suggest that they should be broadly generalizable. The need for autonomy appears to be a species-typical feature of human psychology, and one that emerges early in development.
c) The result you quoted isn’t the effect of interest. The very next sentences are:
“We found a significant interaction between policy and meeting number (i.e., week): OR = 1.14, 95% CI 1.0 to 1.3, P = 0.025. Simple effects tests revealed that attendance in the mandatory groups declined over the course of the semester: χ2 = 13, OR = 0.86, 95% CI 0.79 to 0.93, z = −3.6, P < 0.001. However, it was stable in the optional-mandatory groups: χ2 = 0.37, OR = 0.98, 95% CI 0.91 to 1.1, z = −0.61, P = 0.541 (Fig. 1) [attached to this post]. The pattern emerged across all sections, and we observed no interaction between policy and TA (all Ps ≥ 0.397). These results were robust across alternative model specifications (see “Alternative Models” in the Supplementary Materials).”
To put the size of this effect into context, by the end of the semester students in the optional-mandatory sections were about twice as likely to show up to class than students whose attendance had been mandated. I think it’s really interesting to think about why allowing students to mandate their own attendance has such a big effect on their behavior… Whether a student has been subjected to the ‘mandatory’ policy involuntarily or has subjected herself to it voluntarily might seem irrelevant to the question, “Should I go to class today?” Voluntary or not, she faces the same options, and her options promise the same rewards with the same probabilities. Nevertheless, if she is like most people, then opting into the policy voluntarily could certainly influence her attendance and much else!
Hi Simon, my comments were more aimed at the OP characterisation, commenters and causal readers, given there’s a tendency for results like these to be over/misinterpreted by people with a certain disdain for common teaching structures, rather than your paper itself.
RE (a) I realise pre-commitment alone wasn’t the sole effect, my point was just that nor is it autonomy alone, hence not a lesson in favour of autonomy generally. Perhaps ‘autonomously chosen pre-commitment’ captures the interaction. RE (c) I was aware it wasn’t the effect of interest, but thought it important to mention because ‘twice as likely to show up in class by end of semester’ sounds much more impressive than ‘statistically indistinguishable for most of semester and then distinguishable at the end’, and yet the latter seems like an equally accurate description that potential adopters should be aware of when forming their expectations.
I think your characterization of our results is a bit misleading. “Around twice as likely to attend by the end of the semester” accurately reflects the (large) cumulative effect of two diverging trends. It’s not just about the end point, but about the consistent pattern throughout the semester.
Saying that the groups were “statistically indistinguishable for most of semester and then distinguishable at the end” misrepresents how to interpret results from a mixed model in a repeated measures design. By modeling the covariance structure of the repeated measures, the model can detect consistent effects over time that might not be apparent when looking at individual time points in isolation.
The interaction reflects that attendance in the mandatory group steadily declined over time, while remaining stable in the optional-mandatory group. We probe that further with the simple effects tests presented in the results. I hope this clarifies how to interpret the results!
I wonder how the study would play out in different cultural contexts, with the race and gender of the educator, and whether there would be any long term effects on job satisfaction. I could imagine that in more conservative states, students would interpret giving them such autonomy as somewhat frustrating and the instructor not really knowing what they are doing, especially if the instructor is a minority and underrepresented educator. I could also imagine students growing up with low tolerance for being in low autonomy situations, which is the case in most employment situations, and therefore may not fair as well in the job market after they graduate. Just some thoughts to further develop these studies.
Thank you for this post. I think it nicely puts things into perspective!
The attendance idea sounds very interesting. I might try it out. I’ve tried all sorts of approaches to attendance (not tracking it at all, relatively harsh requirements, and everything in between) and it’s tough finding something I like. Maybe this is a good solution.
On the topic of autonomy (or at least choice) people might also be interested in Andrew Mills’s “Letting Students Choose: Investigating the Menu Approach to Graded Work.”
If you do decide to try it, I hope you’ll shoot me an email: my last name at simoncullen dot org. we’ve seen generally high levels of student uptake of optional mandatory attendance— generally in line with and sometimes exceeding our expectations. if it’s possible, I’d love to stay in touch with as many people who try this in the fall as possible. That will allow us to better establish the generalizability of optional-mandatory policies! so if you are planning to try it in your own class, please please please let me know.
I have given my students tremendous autonomy for over a decade, so I’ll share my practices and experiences. First, a caveat: I teach management accounting to Executive MBAs, so they are more mature, and though the course actually has a lot of philosophical content (see here), it is also has practical implications for my students’ day jobs. I will let readers judge how much my observations apply to their own students.
I structure my course as an “all-you-can-eat buffet in bite-sized pieces”. The course is divided into six units which are further divided into topics (see the structure of my textbook), 22 in all. There are two primary deliverables relevant to this post: quizzes and discussions. (I use Canvas now). Each topic has a bank of quiz questions, and three associated quizzes: Bronze, Silver, and Gold. They can take Bronze quizzes as many times as they want, keeping the highest score. They can take Silver only 4 times and Gold only twice, but Silver is worth 2x as much as Bronze, and Gold is worth 2x as much as Silver. So there’s a clear strategy here: do the lower value quizzes enough to prepare for the higher value ones. I also have a new Discussion on Canvas running week (typically 7-9 weeks), and students get lots of points for an Original Post that describes how some topic in the course material applies to a situation they care about (typically work, home life, current events, politics, sports, and sometimes situations drawn from fiction). They also get points for commenting substantively on others’ posts. My grading rubric is “a good effort in good faith gets full credit”, because in my experience they care enough about the situations and their classmates’ respect that they do a good job without the threat of a low grade from me.
This structure offers several types of autonomy. First, who eats everything at an all-you-can-eat buffet? It’s not reasonable for students to take every quiz—that would require at least a 4-5 credit course, and mine is only 1.5-2.5 credits depending on the program. So I let them choose which ones they want to do. They can focus on some topics and ignore others, or ignore the Bronze (low-point) quizzes. Second, their discussion posts and comments can draw from any topic at any time. Third, their grade is determined by their point total, so they can trade off (within limits) more quizzes for less discussion participation, or the other way around. Finally, all quizzes are due at the very end of the course, so they can do them whenever fits their busy schedules.
What’s the impact? First, the autonomy causes a lot of students great stress. Every year, especially at the beginning of the course, I get numerous complaints that they can’t figure out what their supposed to do and it’s too confusing. These students don’t want this much autonomy, they want me to tell them what to do. I explain to them why I do this. Some students find topics X and Y really relevant to their work, while others find topics Z and W relevant; some like quizzes and others like discussing applications of the material with their classmates; some have time crunches early in the course, others have them late.
Eventually, most students come around, as they see the value of so much autonomy and get more comfortable with the rhythm of the discussions. As a result, they work WAY harder than under my more traditional methods, and as far as I can tell, learn much more. The first time I did this, students freaked out because they worried *someone* would eat the whole buffet and set an impossible curve. I got low ratings that year, but worked out a deal with the program director for year 2: if students earned 25% more points than what earned an A in year 1, they would get an A for sure. The next year, I had so many A students I had to raise the threshold again…and again…and again. And they gave me my first two teaching awards.
In the end, I’d say the “Millian hope” is partially secured. A lot of knowledge and autonomy does lead to some happiness, but it also causes a lot of stress!
My initial question with things like this is ‘how much more work are you about to ask an adjunct to do, for no additional money?’
Hi ECD, this is one of the reasons we were interested in developing this—unlike my previous work, some autonomy-supportive policies like optional-mandatory attendance require very little extra work for instructors to implement. And giving a number of “freebies” (meetings that students are allowed to miss without permission or penalty) both increases autonomy and cuts down on the amount of negotiation instructors must do when students want to be excused from class! What do you think?
Been a while since I taught anything, as I bounced quite hard off academic work, despite retaining historic and familial ties to that universe, but limiting it to that should help significantly. I think you may still be underestimating the potential increase in work that comes from having multiple different requirements for the same (especially larger) class sizes.
Theoretically this ought to reduce work as instead of tracking attendance for 30 students (I assume no one is actually physically tracking attendance for truly large class sizes), I’m only tracking it for the 13 who opted in to mandatory attendance. In practice, I’m less sure on effect, though I can’t put my finger on why besides a generalized, probably unfair, resistance to change.
I could come up with a story about lack of consistency requiring more mental effort/breaking the class into groups, but so long as it’s limited to attendance, I don’t think I’ve got a real beef. The big list of ‘examples of opportunities to increase student autonomy’ however makes me nervous–and if I were to steelman my ‘resistance to change’ position above it would be that educational fads tend not to end up being limited to low hanging fruit and do tend to create significant additional work for teaching staff.
Alternatively, there may be something to the argument that the more ‘mandatory’ requirements you open to individual selection, the more ‘mandatory’ requirements the students will expect you to open to individual selection–but this and the above are slippery slope arguments rather than actual disagreement with the proposed changes to attendance policies on their own.
I’ve used optional-mandatory in my 200-student course that has lectures, 10 discussion sections, and 4 problem-set sessions. I use Acadly (an app that uses bluetooth to check that students’ phones are in the same room as you) to manage attendance. Once you’ve got it set up, it works well!
I let students select policies for each course component individually—they register their preferences using a Google form after the first discussion section (which I paternalistically require them to attend—I want them to at least make informed decisions and they never seem to mind!)
A real low-hanging timesaver is creating a “Syllabus Bot” that answers student questions/emails that are already covered in the syllabus—it makes a huge difference in a big class! You can chat with the one I did a year ago at poe: https://poe.com/80101-Syllabus-Bot (you could create a custom GPT that does the same thing!)
I don’t mean to go off on a tangent, but Acadly sounds interesting but I fear students complaining about their grades in a attendance choice scenario, possibly complaining that the app was not appropriately set-up and failed to track their attendance. Would you Mind sharing your set-up procedure?
A ChatGPT bot to answer syllabus questions is very awesome!
First off, I want to say that these suggestions are well thought-out, interesting, and reasonably plausible. I am even tempted to adopt something like the extension policy you suggest.
I do, however, want to voice one concern. I’m already troubled by the extent to which college already encourages the kind of strategic thinking which your policies promote. To my mind, all the thinking that students do about how to game course policies so as to maximize their GPA, or their GPA-to-effort metric, is a distraction from course content. Before you even present your autonomy-enhancing syllabus, students have already selected their courses strategically. And as you know, many students opt for philosophy classes as part of a plan to maximize their grades in the courses that are relevant to their future employment, such as their STEM courses, often at the expense of their humanities courses.
I try to design my courses in a way that does not encourage strategic thinking. To the contrary, I try to communicate clearly to students that gamesmanship is just a distraction from what really matters, namely learning. I do this by conveying that there is a surefire way to do well in my courses, namely showing up, and learning the material. No complex incentive systems, no frills, and no distractions.
What a great topic for this week, as the Biennial Workshop-Conference of the American Association of Philosophy Teachers begins at Otterbein University! Cheers to the excellent discussion of student autonomy.
There has been a lot of recent interest (much of it workshopped at recent AAPT meetings and AAPT co-sponsored Teaching Hubs) in specifications grading, which is a constellation of approaches to syllabus design that embrace student autonomy. Interested folks should check out two essays:
Dennis Earl, “Two Years of Specifications Grading in Philosophy.” Teaching Philosophy 45 (1):23-64 (2022). (See here.)
and
Sarah Vitale and Dave Concepción. “Improving Student Learning with Aspects of Specifications Grading.Teaching Philosophy 44 (1):29-57 (2018). (See here.)