Is there a German word for a feeling that combines admiration, weariness, and a touch of disgust? That word would be handy as we continue to catalog attempts to teach in a world of artificial intelligence, such as this one from professor of technology and business Panos Ipeirotis (NYU).
As students are increasingly tempted to use AI to read assigned texts and write assigned papers, many professors are increasingly desperate for ways to ensure their students are actually learning.
A popular recommendation is the oral exam. Have a conversation with the student in an AI-free environment in which their learning can be assessed. It sounds like a good option, and it often is. (I’ve used them myself over the past few years.)
But the problem is that they don’t scale. If you have too many students, you won’t have enough time to give oral exams (and certainly not enough time to give each student multiple such exams during the course, which is probably much more effective at incentivizing learning than just one).
What to do? Here’s Professor Ipeirotis’s idea: have an AI conduct the oral exam!
And so he developed an AI examiner, and tried it out:
We used ElevenLabs Conversational AI to build the examiner. The platform bundles the messy parts (speech-to-text, text-to-speech, turn-taking, interruption handling, …) into something usable. And here is the thing that surprised me: a basic version for a low-stakes setting (e.g., an assignment) can be up and running in literally minutes. Minutes. Just write a prompt describing what the agent should ask the student, and you are done.
Two features mattered a lot for our setup:
-
- Dynamic variables: pass the student’s name, project details, and other per-student context into the conversation as parameters, to allow personalized exams
- Workflows: build a structured flow with sub-agents instead of a single “chatty” agent trying to do everything
We ran a two-part oral exam.
Part 1: “Talk me through your project.” The agent asks about the student’s capstone project: goals, data, modeling choices, evaluation, failure modes. This is where the “LLM did my homework” strategy dies. You can paste an assignment into ChatGPT. It is much harder to improvise consistent answers about specific decisions when someone is drilling into details.
Part 2: “Now do a case.” The agent picks one of the cases we discussed in class and asks questions spanning the topics we covered: basically testing whether students absorbed the material or just showed up.
To handle this structure, we split the exam into sub-agents in a workflow:
-
- Authentication agent: Asks for the student’s ID and refuses to proceed without a valid one. (In a more productized version, we would integrate with NYU SSO instead of checking against a list.)
- Project discussion agent: Gets project context injected via parameters. The prompt includes details of each project so the agent can ask informed questions. The next step is obvious: connect retrieval over the student’s submitted slides and reports so the agent can quote and probe precisely.
- Case discussion agent: Selects a case and runs structured questioning. Again, RAG would help with richer case details.
This “many small agents” approach is not just aesthetic. It prevents the system from drifting into unbounded conversation, and it makes debugging possible.
Each exam generated a transcript which was then graded by a “council of LLMs”:
Three models (Claude, Gemini, ChatGPT) assessed each transcript independently. Then they saw each other’s assessments and revised. Finally, the chair (Claude) synthesized the final grade with evidence.
And how was the grading?
The grading was stricter than my own default… The feedback was better than any human would produce.
You can read all about the experiment here, including what Ipeirotis learned from it, what the students thought, and what he’d do differently next time. You can also try out taking the exam.
Discussion welcome.
Congratulations on losing your job.
Agreed. Fighting AI with AI is not fighting AI at all.
It depends on what the goal is. If you think AI is inherently bad and needs to be defeated no matter what it does, then you’re absolutely right.
If you think that AI is one tool among many, and that it can be used in good ways, but that a lot of its use by students short-circuits their learning, then there might be totally reasonable ways to use some AI applications to ensure that students don’t use other AI applications to short-circuit their learninn.
and trying to lose our jobs!
If we want to resist losing our jobs to machines, we need to be able to explain why it would be a bad thing for us to lose jobs to machines. We need to be able to explain what society loses without us, or with fewer of us.
It might not be easy to argue for this in a way that convinces many, but a lot of what society loses is simply… us. There is a positive value to learning from a human. A teacher isn’t simply a teaching technology, possibly replaceable by a superior technology. A teacher is another person, and that’s important in itself–important, I mean, for the student.
Cf. Nic Eth VIII-IX.
It strikes me as very likely that here are advantages to learning from another person. If these advantages exist, they need to be described and demonstrated.
Not necessarily. Compare playing music with a real person to playing music with a computer program, or seeing a great artwork in person to seeing a mechanical reproduction of it. In each case the first thing is better, but it would be unduly concessive to think that we need to have a well-worked-out answer to the (hard!) question why the first thing is better before confidently asserting that it is. “If you’ve got to ask, you ain’t never gonna know” does more justice to the matter.
If we want to keep our jobs, the advantages humans have over machines need to be described and demonstrated. Likewise, when we decide which activities it is most important to keep in human hands, the advantages humans have over machines will need to be described and demonstrated.
With respect to deciding which activities it’s most important to keep in human hands—yes, certainly.
But with respect to keeping our jobs, I think it’s much less clear, in at least two dimensions. First, it’s unclear how much influence any attempts we make at persuasion could actually have (for the record, I’m not especially optimistic); second, insofar as any of our attempts at persuasion could have some influence, it’s unclear which forms of persuasion would be most effective in practice. And it does seem to me at least possible that reminding people that they know there’s a difference—indeed an absolutely vital one—and they don’t need to be able to articulate that difference to know just how vital it is, may well be a more effective form of persuasion in practice than pretending that the existence of a difference isn’t obvious to thoughtful and humane people. (And, also for the record, if—in what again strikes me as the much more likely scenario—our efforts at persuasion can’t actually make much of a difference, this way of representing matters at least seems to me to be doing more honor to the facts.)
Isn’t all (philosophical) teaching really a kind of reminding, though?
Still it’s good to be reminded of that.
Why is the first case better? Too dogmatic of an answer. It feels “right” that a painting IRL is somehow better and then draws an analogy from that to that a human tutor is better than an AI
Completely agree. Extracting from instruction the emotional connection that almost always exists in human interaction is ridiculous. Personality is inherently part and parcel of conveying learning–remember your own favorite instructors? Surely part of that is who they are and what that meant to you.
This is true for some modes of learning, but certainly not all. There’s also lots of learning, particularly the learning that we do when researching, that involves interacting with the thing you’re learning about but not with other people.
Some learning works better through personalities. Some works better without it.
One thing that jumped out at me was the post-survey data, in which students said they didn’t like this protocol. On my reading of the literature and based on my experience, effective teaching and deep, sustained learning require robust interpersonal relationships. I’d be interested to know more about why the students had the reaction they did. Is it simply because they don’t like the oral exam format? The AI voice? The researchers note that the students also said that the protocol tested their knowledge. But we might wonder whether the trade-off is worthwhile: more reliable tests of knowledge versus better development of relationships.
I’m reminded of things I’ve read recently that highlight an increasingly transactional view of college. In addition to worries about hypocrisy and undermining one’s own employment, I think we should consider about how our reactions to the arrival of gen AI is feeding into things like eroding the interpersonal nature of higher ed. This strikes me as related to things discussed on this blog previously, such as surveillance.
I think this satire of academic philosophy is a bit too on the nose.
Really interesting! For anyone interested, this connects closely to a preprint my coauthors and I posted on using LLMs to simulate a viva voce as a tool to support assessment and academic integrity. The paper reports a proof-of-concept “virtual examiner” that asks probing, submission-specific questions and outputs a transcript plus an assessment/confidence score for a human examiner. It also discusses practical deployment constraints. For example, you’d likely want this in a proctored/secure setting, and you’d need to guard against people trying to sneak hidden instructions into the submitted text to manipulate the examiner.
Preprint: https://arxiv.org/abs/2511.05530
This is obviously happening, and I doubt it’s the most dramatic change AI will bring (if we can maintain the required global infrastructure in the years to come).
Nope.
Individual meeting assessments (oral exams) do scale! I did two 20-minute ones in the fall, for four sections of about 20 students, and absolutely loved it. Here‘s a rough account of what I did.
A couple of keys to scaling:
I couldn’t say whether this took more or less time than discussion forums. But I’m actually not worried about it. If it did take a bit more time (and I’m not saying it did), I’d rather spend a little more time doing something fun and rewarding than a little less time doing something… less fun.
Yes, I have made this point several times before in DN threads—the ‘oral exams are not scalable’ claim is uncritically accepted by many. A typical philosophy course has two essays which take about 20 minutes per student for the instructor to mark. Continuing with the status quo, where many students are submitting AI-generated essays, is a waste of everyone’s time. A better option is for the instructor to only set one essay, which they spend 20 minutes marking, and then follow this up with a 15-minute oral examination that tests each student’s understanding of the essay they submitted. This disincentivizes students irresponsibly using AI without any increased time burden on the instructor.
Also, this is a non-problem:
Simply set up a calendar at the start of term with your available times for student to book their slots by a cut-off date and don’t allow changes after that date unless students have appropriate documentation showing why they can no longer attend the slot they booked.
I’m curious if you think there are some particular virtues for the essay follow-up being an oral evaluation. I’ve been thinking I would simply add some individualized questions to in-class exams based on submitted papers.
I make the case for take-home essays plus oral exams here – https://ipscommons.sg/reimagining-academic-assessment-in-the-age-of-ai/. I’m not sure that individualized exam questions can match a back-and-forth conversation in testing the degree to which a student has taken intellectual responsibility for the work they submitted, but I would have to hear more about how you planned to do it.
I think the general public is getting the message that lots of instruction / assessment can be automated. People are using AI to teach themselves all sorts of skills and subjects.
And universities are under financial pressures from declining enrollment, declining external funding, etc. Plus there are growing demands for cost-effectiveness and efficiency from trustees and legislators, as well as from people who can’t afford high costs for post-secondary education.
Pretty much any existing undergraduate philosophy course can and probably soon will be entirely automated. Our departments will be restructured, severely reduced, or eliminated entirely.
I think the only way for philosophers to survive this is for us to really start intensively engaging with AI. This includes things like building specially-trained or engineered philosophy AIs (e.g. working on making AIs that are experts in abstract reasoning). And things like working on problems associated with AIs (e.g. alignment, ethics, AI truthfulness, etc.). And there are probably lots of other was to intensively engage with AIs.
Or you can resist, and celebrate your purity, until you get a pink slip letter from your Dean. I’ll bet that letter is being drafted right now.
Self-interest is a powerful drug.
“There is a fascist government that is oppressing pretty much everyone and destroying pretty much everything. The only way to survive is to intensively engage with this government. Engineer fascist philosophy. Things like working on fascist problems. Anything less is a celebration of fascist-free purity.”
And overgeneralization is a fallacy.
And your conclusion doesn’t follow from your premises.
It’s pretty funny that you think that an extremely difficult thing (teaching philosophy) will soon be automated, but also that the alternative activities you list won’t be. All of the “intensive engagement” you list can be done… by AI.
Indeed, It is far easier for AI to write a paper on the alignment problem or truthfulness than it is to literally teach a class. It already does this.
You’re letting yourself believe that there are alternatives for humans because it allows you to avoid seeing AI for what it is. It doesn’t just selectively colonize some human cognitive activities, letting you clever in-the-know types keep their cognitive jobs. It colonizes all of them.
If you’ve got a better idea, one that will persuade admins, trustees, and legislators to keep funding philosophy, let us know.
Maybe try to educate them?
About what? About the advantages people have over machines in doing the job?
About what cognitive offloading is, about what it means to say that machine learning systems ‘colonize human cognitive capacities’ (to pick up phrasing from an earlier comment in this thread), about the downsides as well as the upsides of the pattern recognition capacity of LLMs, about the long range social implications of widespread integration of LLMs into everything we do, about the responsibility of educators (including admins) to long-term thinking, not just the short-term thinking esp. when it comes to radically disruptive technologies.
When the time comes to cut costs, admins have shown again and again (as documented in these very pages) that they will reduce or eliminate philosophy programs (along with other humanities programs). And we’ve usually responded, as you responded, by pointing to vague issues which are irrelevant in any cost-cutting scenario. Not a single philosophy department has been saved by the points you mentioned. We face an existential threat, and we need effective ways to counter it.
“The feedback was better than any human would produce.” If that’s true, what justification could we have not to use this?
Humans use LLM to generate text, so I think that will be false for a while.
I think so to, but that’s an empirical question. If the feedback is, or becomes, better, what justification could we have for not using this?
Call me old-fashioned, but I think an experiment like this should need IRB approval, and I feel badly for the 83% of students who were stressed out by it.
Weirdly, IRB approval is needed for trying out new things if you want to publish the results, but it’s not required for just trying out new things without publishing the results. That’s probably really important for everyone who tries new things in any class they’re teaching, but it’s definitely an odd limitation on publishing the results of your new teaching methods.
Interesting experiment.
I suspect the role of inter-personal relations in university-level learning (even in large lecture courses) is under-appreciated.
If I were an 19 year-old student in this scenario I probably would feel disrespected. Also unsafe (conscious of the potential for privacy violations, and of my teacher enabling surveillance society norms to develop through this set-up).
Or more realistically, I’d have some kind of inchoate aversion to the examination, without being able to name it. (How could I name it if everyone around me is pretending to think, or really thinks, that every new tech app is a good app?) Or maybe I would just find examination by LLM amusing and life absurd, which would have the effect of shifting my whole experience of the course, regardless of its content.
If they work as suggested, then AI oral exams are superior to the business-as-usual approach where students are assigned take-home essays with no disincentive to cognitively offload the whole thing to AI. However, I still think that instructor-student oral exams are the superior option.
As several commentators above have noted, interpersonal connection is an important part of learning. When I conduct oral exams with my students, I engage with them personally and show my genuine interest in understanding how they are thinking through the topic. I find that students react very positively to this and value the one-on-one interaction. I doubt that an AI chatbot could replicate this.
Furthermore, as someone who regularly experiments with the latest AI models, I am not convinced that they can match the philosophical insight that me and my colleagues bring to a philosophical discussion. What I’ve always found most thrilling about philosophical conversations is when the interlocutors think on their feet and show creativity in working through a line of reasoning or analysis that they have no prior familiarity with. This is distinctly different from the very polished and knowledgeable, yet regurgitating persona I get from LLMs when discussing philosophy with them. As my teachers inspired me, I hope to inspire my students by bringing this kind of philosophical insight to our conversations. Maybe I am naïve about this, or maybe some AI models will soon match and then surpass human philosophers for philosophical insight. However, for the time being, this seems like one of the important things I add as an educator.
Regarding the supposed scalability problem with instructors conducting oral exams, see the rebuttals above. However, I would also add that universities need to invest in more one-on-one interaction between instructors and students as part of their response to the threat AI poses to their business model. And, if they do this (my university has already made some steps in this direction) it would also make conducting oral exams much easier.
Finally, the author suggests that AI oral examiners may be less biased and more objective than human examiners. This is an interesting claim that deserves further investigation. One of the problems with oral exams is that they can be overly performative. Even if the goal is to narrowly assess some intellectual attribute, the charisma, rhetorical finesse, and charm of the student, or their lack thereof, can unfortunately influence their assessors. When I use oral exams to test whether a student has taken full intellectual responsibility for their submitted essay, I have a procedure that helps to reduce the influence of these things. Rather than marking the student from scratch with the full range of marks on offer, I use the mark I awarded their essay (which I mark blind) as their default and maximum mark. If a student got 80% for their essay then they start the oral exam with a mark of 80% penciled in for it. By the end of the interview, if they have been able to adequately answer all of the questions they were asked about their essay, thereby demonstrating they have taken full intellectual responsibility for it, then their essay mark is replicated as their oral exam mark. On the other hand, each observed failure of intellectual responsibility during the interview leads to a reduction in the oral exam mark. This helps me zero in on what I want to test and ignore irrelevant attributes. A student may come across as very nervous, lack the fluidity of someone who speaks English as a first language, or be an extremely unengaging speaker. Yet if their answer to each question shows they fully understand their submitted work, then their default mark is unchanged. By contrast, a student whose gaps of understanding are clearly evident will be marked down and no amount of charisma and charm can escape this.
I second the point about the value of interpersonal interaction between students and teachers.
On LLMs matching human philosophers: human philosophy teachers are still better at informal reasoning in natural languages. While LLMs are better than most of us at formal proofs in symbolic languages, from what I’ve seen, they struggle to grasp the structure of an argument given to them in ordinary English. They can’t distinguish between two arguments for the same conclusion vs. two premises that are part of one argument (“co-premises vs. independent arguments” in an argument map) or inference objections vs. objections to premises, and they’re also bad at not randomly adding extra premises or other commentary. I’ve given my argument mapping quizzes to ChatGPT and it tends to score in the F to C range. I give examples here.
There’s another reason why it’s important to teach students to reason themselves, rather than teaching them to ask an LLM to do their reasoning for them: while people will use LLMs to improve some of their decisions, particularly by gathering more information, inevitably, they’ll do the vast majority of their reasoning in their head and in conversation with other people. Moreover, even when they get information from an LLM, they’ll infer conclusions from this information on their own.
This isn’t a small thing that we humans are better at. Helping our students become better at reasoning (thinking critically) might be the best thing we college teachers can do for them.
Someday, LLMs might catch up to us in informal, natural language reasoning. If that happens, it be a game-changer in education and probably everything else. But they’re not close yet and we’re not in a position to infer that they’ll ever get there.
Our experiences concur with yours regarding LLMs performing poorly at argument mapping. Indeed, I believe that our critical thinking course is the only course left at our university that still uses a take-home exam, which we are able to use because LLMs can’t handle the argument mapping questions and consistently fail these exams.
I find this somewhat hard to believe. I’m not a fanboy of AI and have severe doubts about some claimed capabilities, but in my experience, various LLMs at the top tier have done great at argument reconstruction and mapping for over a year now. Have you tried the top-tier models with “thinking” enabled, like Anthropic’s Claude Opus 4.5? I would be very, very surprised if first-year students are consistently and obviously beating that model. But stranger things have happened—I don’t mean to suggest dishonesty on your part!
Yes, we have and we found that top tier models perform poorly at argument mapping. I’ve even sent some of our questions to an acquittance who said they had good results getting LLMs to argument map and they could not get the models they used to produce the right answers. If you are interested in putting this to the test then please email me and I’ll happily share some of our questions with you and eagerly wait to see if you can get a better result from top tier models.
Thanks! That’s helpful, and, I confess, surprising but welcome news!
I believe fighting Ai with Ai can be beneficial. Within this experiment, I found it crucial how material taught in person is part of examinations. Oral examinations would take too long if the class was to be on the bigger size. Perhaps within classes of 20 students or less, oral exams regarding material that should have been learned could be a good idea. I believe implementing more practical examinations could be a better form to deter students relying on AI. In my college, we are given a semester long project that requires us to reach out to experts in a given field and work alongside them to create a solution for a given issue. At the end of the semester we are asked by a group of experts about how we went about reaching said solution and they ask questions to verify if we really understood what it was we just presented.