How, as a researcher, can you use AI tools like ChatGPT in a way that doesn’t compromise your integrity, creativity, and independence?
In the following guest post, Jimmy Alfonso Licon (Arizona State University) explains how he does it, laying out how he approaches his writing process and the roles he assigns ChatGPT in it.
As Dr. Licon describes it, his “resulting workflow is neither purely human nor AI-written.”
Those who make use of AI in their research and writing are encouraged to share their methods and processes in the comments. Also of particular interest is how human-AI “hybrid” works should be treated, institutionally (by journals, universities). Discussion welcome.
(A version of this piece first appeared at Dr. Licon’s newsletter, Uncommon Wisdom.)
Have Pen, Laptop, and ChatGPT, Will Publish
or How I Use AI Without Sacrificing Creativity and Independence
by Jimmy Alfonso Licon
People sometimes imagine writing as a flash of inspiration, a heroic sprint at the keyboard, and then a finished paper. My own process is considerably less cinematic and much more modular. It involves a pile of printed articles, a pen, a computer, and a large language model. Each plays a specific role. And together, they help me turn a half-formed idea into a shareable, defensible piece of scholarship.
I usually start with a nagging thought. An irritation with a familiar argument, or a pattern I see across different debates, or a question that just won’t leave me alone—something that continues to bug me. At this stage, I begin by writing down a very rough abstract: a paragraph or two sketching the core claim, the basic structure of the argument, and why it might matter. It is only meant to capture the rough intuition. Nothing beyond that. The point is to get the idea out of my head and onto paper where I can see it, poke at it, and at some point, ask questions about. The three main questions I ask are: is the idea genuinely novel? Is it interesting enough? Is it intellectually defensible? The answer must be affirmative in each case before I proceed.
So for the next step, I hunt down the relevant literature by asking ChatGPT, surfing Google Scholar, and asking colleagues who work on similar stuff. That means scanning databases, following citations, and running it by ChatGPT, prompting it to analyze the idea like a referee at a top journal. If I find that someone has already the same article—or something close enough—I will usually shelve the idea. Sometimes, though, it means shifting the focus, narrowing the scope, or locating a gap or tension in the literature. The goal here is to avoid writing something redundant.
Once I know there something new, defensive, and interesting to say, I print out the related research. This is probably the most old-fashioned part of the process. I still prefer to read serious work on paper. I mark it up with pen and highlighter—different colors for different purposes like central arguments, key definitions, clever examples, potential objections. The marginalia are often more important than the original text for my later writing when I consult them for questions, counterarguments, little arrows between sections that should really be read together. By the time I am done, I have an excellent sense of how the idea fits into the broader literature.
Alongside the printed articles, I keep a dedicated notebook. Each paper I read gets its own entry: author, title, main thesis in a sentence or two, and then a set of bullet points keyed to page numbers. In this part of the process, I try to capture what will matter for my argument. Sometimes that means summarizing a section. Sometimes it means writing “this seems wrong because…” and then a few lines of reasoning. The notebook becomes an index of the conversation I am joining, but organized around my project rather than the order in which I happened to discover the sources.
Only after I have done that analog and cognitive legwork do I bring in ChatGPT. At this stage, I treat it as a kind of overcaffeinated colleague who has read a lot, reasons quickly, and is perfectly happy to brainstorm for as long as I like, but whose judgment I do not automatically trust. I feed it the rough abstract, along with summaries or quotations from the key articles I have been working through. Then I ask questions that I would normally ask another philosopher over coffee: how do these pieces fit together? What are the obvious objections I haven’t considered? Are there surprising connections between these arguments that I have overlooked?
The aim here is to use a different kind of approach to shake loose alternatives I might have missed. Sometimes the model tells me things I already know and other times it gives helpful suggestions about framing, distinctions, or other counterarguments I had not fully articulated for myself. I jot those down into the same notebook, flagged as “AI-suggested” so I can keep track of what came from where.
After that, I ask the model to write a rough synthesis of the project on the basis of the abstract and my summaries of the research. Think of this as commissioning a quick, imperfect mini-essay from a smart but hasty graduate student. It tends to get the big picture more or less right, but the details are often off, the nuance is usually thin, and the voice is certainly not mine. That is fine. The point is to see my own materials in a different light to highlight which parts of the project are doing real work and which are just taking up space because I happen to find them interesting.
And then, using the rough abstract, my marked-up articles, the notebook entries, and the synthesis as a kind of mirror, I draft a detailed outline. For a paper-length project, that outline can run to several pages. I break the argument down into sections and subsections such as introduction, motivation, existing views, critique, positive proposal, replies to objections, conclusion. Under each heading, I list the claims to be defended, the sources to be used, and the transitions I need to make explicit. If the AI synthesis suggested a helpful ordering or highlighted a missing step, that gets incorporated here, but in my own words, and only after thinking through it again.
Drafting itself is slower and more solitary. I write on the computer, but I keep the outline and my notebook next to me. The first full draft is where the project becomes recognizably mine. I write until I have something that is complete enough to criticize but still loose enough that I do not feel overly attached to any particular sentence. And then I put it away for a bit to work on something else.
Letting a draft “mature in my brain” is part of the process and allows me to start on another project, while my mind works on the draft in the background. Often I will realize, in the shower or on a walk, that a section belongs earlier, or that an example I used in class would work beautifully here, or that the conclusion needs to be less triumphalist and more modest. I usually jot those insights down as they happen in a notebook or they will be lost forever. And then, when I return to the text after a few days or weeks, I see it with fresher eyes.
Only then do I involve ChatGPT again, and now in a very different role. At this stage, I ask it to behave not like a chatty colleague but like a demanding referee. I paste in the draft and explicitly request a rigorous, critical report: identify unclear passages, gaps in the argument, undefended assumptions, missing literature, distracting tangents, and so on. I encourage it to be uncharitable, and to read the paper like an overworked, skeptical reviewer might. Often, but not always, it will spot real weaknesses in the paper like the need for a distinction, firmer argumentation, or including a source I forgot about.
Crucially, I do not accept these suggestions wholesale. This is where experience and judgment matter. I go through the “referee report” line by line. After a few rounds of rereading and revising, I arrive at what I consider a “shareable draft.” At that stage, I rely on conference presentations or having a colleague look it over, and then eventually I submit it to a journal.
The resulting workflow is neither purely human nor AI-written. It is a hybrid that leans heavily on traditional scholarly virtues like careful reading, slow note-taking, attention to existing work, and a willingness to let ideas ripen over time. The tools—pen, computer, ChatGPT—are simply different tools for extending and scaffolding my limited memory, pattern recognition, self-critique. What matters, in the end, is that the argument I sign my name to is one I understand, endorse, and can defend without any of those tools in the room.
Related: “Ethics announces AI Policy“, “The Ethics of Using AI in Philosophical Research“
You do you. But, the extended mind actually atrophies it.
Actually, if you really believe in the extended mind, then at most what is being atrophied, by outsourcing, is biologically grounded intelligence; the ‘mind’, extended, will include the technology one is densely coupled with (at least on Clark/Chalmers style extended cognition). That said, it’s a substantive question when and under what conditions (if extended cognition is true about the metaphysics of the mind) usage of AI is bona fide cognitive extension as opposed to something that (e.g., by Clark’s glue and trust conditions) falls short of it.
Genuinely extended cognition (Clark Chalmers style) can still atrophy the mind. One example of that would be cases where extending the mind via technology atrophies the brain in such a way that the extended version, brain plus tech, is worse than the unextended version, on whatever measures you’re using.
I don’t think anyone denies that. Every time you ask a friend for help (particularly if they’re an expert that you’re deferring to) you’re letting some of your skills atrophy. But in a lot of cases, this kind of social (or technological) scaffolding can enable you to do things you wouldn’t be able to do otherwise, and can let you develop other skills that would have atrophied without this support.
I think a lot of people are too willing to let their spatial navigation skills atrophy and just offload all navigation to a GPS system – I prefer to at most get an initial route from the GPS and then study it and then try to re-create it (or a modified version) myself, and only pull over and ask again if I think I missed something. Other people probably think I’m too insistent on relying on my own navigation, and that might be right – perhaps for most people the best balance is somewhere in between.
Setting aside ethical issues in using current models that are based on wholesale theft without credit or compensation, there is a serious issue of de-skilling and cognitive atrophy. I strongly discourage students from using LLMs to replace parts of their own thinking since the cost, long term, is high.
It’s worth actually reading through his description of what he does. Most of his use of ChatGPT doesn’t involve replacing his thinking. In some cases, it replaces the use of Google Scholar, or conversations with a colleague, which are also worthwhile skills to keep honed. But I don’t think there’s much, at all, actual thinking that is being offloaded in this described workflow.
Even if you think there is value enough in the final written product regardless of how it was produced, there is also significant value in the fact that this product is tied to a human enterprise, much of which is being severed in your vision. There is enormous value, not accidentally related to praiseworthy thinking and writing, in being able to find sources yourself, evaluate them, learn how to spot weaknesses over long and grueling hours without asking a mindless machine to shortcut it for you. Yeah yeah, experience and judgment are still important. But how is sharp judgment going to survive when scholars don’t actually do all this work themselves? Are you recommending this method only for senior members of the field?
Maybe in the end you produce something containing valuable philosophical content, but there is also something worthy of some degree of revulsion here.
I worry that this line of reasoning overgeneralizes, and in a way that threatens to condemn a wide range of uncontroversial epistemic shortcuts.
If I want to know the time, I can do so quickly and reliably by looking at my watch or phone. I could instead refuse this shortcut and construct a sundial, learning a great deal about astronomy and geometry along the way. But it would be strange to think that using a watch involves a significant loss of value—still less something “worthy of revulsion”—simply because it bypasses a richer human practice. The fact that habitual watch-use may leave me unable to build a sundial does not by itself count against using a watch when my aim is just to know the time.
Similarly, if I want to get from Los Angeles to New York, I can fly, or I can walk for a month, battling weather and fatigue and gaining hard-won experiential knowledge. Choosing the former certainly “severs” a human enterprise in your sense, but again, it is unclear what morally or epistemically significant loss is incurred relative to the task at hand. The point of travel is arrival, not the maximal cultivation of endurance or navigational skill.
These cases suggest a general lesson: the mere fact that a tool short-circuits a demanding human process does not, by itself, show that its use involves a problematic loss of value. Sometimes the relevant good is the product or outcome, not the preservation of every upstream practice that could have produced it.
If writing and research are importantly different in kind, then that difference needs to be identified. Is the claim that certain epistemic virtues can only be developed by doing all of the upstream labor oneself? Or that the point of scholarly writing is not primarily the production of insight, but the cultivation of those virtues? Or that there are distinctive risks of dependence here that do not arise in cases like watches or planes? Without such an account, the objection risks condemning not just AI-assisted writing, but a great deal of ordinary and widely accepted epistemic outsourcing.
GPTZero says your comments were 100% AI generated.
(Yes, AI detectors are fallible, though GPTZero seems to be the most accurate one today. No, I don’t rely on them to catch AI cheating.)
Here’s a discussion that explains why “nobody wants to read AI”:
So, for many readers here, you’re participating in this discussion in bad faith.
Maybe you’re trying to make a point that AI writing can result in useful comments. Or maybe not, since that point would be better served if you were transparent about AI use, e.g., admitting to it at the very end of your comments.
Now we’re left to wonder whether you really believe what you posted here and if you could’ve even arrived at that argument on your own. Either way, there’s a good reason why people don’t or wouldn’t want to engage with AI-written comments…
Independently of the substantive issues, I really object to using AI detectors this way. Can’t you just tell by yourself (and then keep to yourself under the principle of charity) whether it is AI-written in most cases?
To be charitable, I’ll suspend my rule of not replying to anonymous people here:
Yes, I can tell by myself, and then I turn to an AI detector to corroborate a suspicion (but never rely on it as a smoking gun).
But how exactly does the principle of charity apply here? Whom would I be acting uncharitably toward: the human who copy-and-pasted, or the AI that wrote the comments?
If someone is participating in bad faith or otherwise does something wrong, does the principle of charity really require that I don’t call it out?
And how can I make the point that the comments are disrespectful because they’re AI-written, without first calling it out as AI-written?
Not sure why you think it’s important to call me out for calling someone else. If my call-out was an offense, was it really worse than the initial offense? That is, can you not think of any relevant principles that would weigh against secretly using AI to write for others to read?
I remember you quite recently using an AI generated picture without attribution or any disclosure. I agree with Alice’s objection, too.
You mean this one, which was obviously AI-generated (as it matches the style of many other such images), found in the wild (i.e., I didn’t create), and used as a fun/ironic illustration to support a point and not to make the point itself?
Equivocation much?
These are ad hoc distinctions. The same objections you have against a comment that may have been written with the assistance of AI apply to your careless use of the AI image. No one wants to see that content, as you put it. If you choose to call out, be prepared to be called out.
Lol
With human content like that, who needs AI!
Wait, you think that is obviously AI generated? It looks to me like the sort of meme humans have been making for years.
What memes have you been looking at?
It wouldn’t surprise me if it were AI-generated. But it also wouldn’t surprise me if there were a real human artist who is making a point in drawing it in the exact style that slop machines reliably push out.
Sorry, Kenny, I missed your comment here from much earlier.
Yes, if you haven’t been following AI memes, just do a “visually similar” image search for that AI meme I had posted, and you’ll clearly see that trend recently.
And while human artists have use that retrofuturism style before in comics, I’m not aware it’s a recent trend at all. I don’t even think I could find a single human-created example from this century that matches that style.
So, AI is or should be still the first thing to come to mind with those images.
Perhaps it was a bit of irony.
Hi Patrick, I wrote down some quick ideas (framing the overgeneralization line) in response to the post and had ChatGPT clean it up, then I tweaked it so it put things how I wanted. It represents exactly my reasoning – including using the two examples I came up with (sundial and NY/LA) . I’ve seen a lot of pearl clutching about ChatGPT here but not yet a response to the overgeneralization argument against the over-fettishisation of human toil to achieve practical goals. I’ll grant that some kinds of achievements might have intrinsic value and require some kinds of overcoming of obstacles (cf Gwen Bradford’s book on achievement) but the concern is that the reasoning in the post I was replying to overstretched as too broad a critique. ps regardless of what people think about reading ChatGPT text, hopefully philosophers can get use to just evaluating the arguments – this recent paper by Savulescu and Schuklenk is going to age well https://onlinelibrary.wiley.com/doi/10.1111/bioe.70069
Thanks for your accounting, Polly, though that transparency would’ve been much more useful if it had been alongside your initial comments. Of the few norms of that exist today around responsible use of AI in writing, acknowledging AI use is one of them.
This bit seems to get at the crux of the matter here: “hopefully philosophers can get used to just evaluating the arguments.”
Yes, that would be wonderful if we could just focus on substance or content, not on other things, such as lineage, process, agendas, etc.
The problem is there’s increasingly less time to do that, since content can now be pumped out by AI at superhuman rates, in a literal blink of an eye. (We can even assume it’s good content here, as opposed to the slop that it often is.)
So, it can very quickly become a lopsided debate when one side, using AI, can just continually dump content and responses to sink your boat faster than you can bail that water out the boat.
This is related to the “Gish gallop” which we see nearly everyday in US politics now. From Wikipedia:
And so where many people have made the very reasonable and pragmatic choice to not waste their time engaging with those kinds of interlocutors, a similar rationale exists for not engaging with people who are happy to copy-and-paste AI content. It’s basically triage.
(Another rationale, as I already mentioned, is that many people find it disrespectful to be met with AI-written content, instead of interacting with a person who has a real stake in the conversation, i.e., has invested their time and effort.)
The alternative seems to be to use AI to counter AI-written arguments at digital speeds. But what’s the point of that?
It also turns your wish into “hopefully philosophers can get used to AI to evaluate the arguments.” And we’re not going to do that.
If someone tells me that they’ve produced their response with AI I simply will not interact with them further. Because they aren’t in the conversation.
So you objected to an argument that supports certain uses of AI (within limits, I suppose) by using AI? And the objection is that the author used AI to write their argument that supports certain uses of AI?
Yes, to all of this. And there is also value—maybe an even more important value—in an intellectual community defined by collective engagement in, and shared stewardship of, this (prototypically MacIntyrean) practice.
Is it better to type keywords into Google Scholar than to ask a colleague or ChatGPT to list relevant literature? I can see that if you want to evaluate someone’s encyclopedic knowledge of the literature, you don’t want to let them use any of these tools. But are you actually saying that all of these are bad? Or is there some way in which the new technological tool is worse than the old one, or the social one?
My hope is that when most of us run literature searches, we’re not beginning and ending it with just one of ChatGPT, Google Scholar, or even the library’s search tool. Or even just all three of those.
If we are, then that explains a lot about how bad we all are at citing one another, and keeping up with the literature.
I don’t have the time to say everything I want to here, but let me say this:
The bit about using ChatGPT for finding sources is the least of my concerns. However, I sincerely hope that nobody compiles a list of relevant literature either by simply typing in keywords or by asking someone else. I expect scholars to become familiar with relevant literature through a combination of becoming immersed in widely cited sources and conducting a broader search using their judgment to sort the better and/or more relevant sources from the worse and/or less relevant ones. This is something you would have to do even if you first asked ChatGPT to produce a list, so I am genuinely not sure what advantage AI gives you here.
I could tell this thing was written using AI from the first sentence. Absolutely soulless, reads like advertising copy, would only ever be recognized as “good writing” by a bad writer. “Still loose enough that I do not feel overly attached to any particular sentence” is the kind of thing that can only be uttered by someone who views writing as purely technical and a bit of a slog rather than the crucible in which thought is forged.
What a time to be alive. What a time to be finishing a PhD and embarking upon an academic career. Students writing papers using ChatGPT, professors marking those papers using ChatGPT, alleged humanists writing op-eds about how one can write using ChatGPT without sacrificing creativity, independence, or–another word that should have been invoked, but whose perdurance is even more dubious–integrity.
I will hold a lifelong grudge against every weak soul who participated in turning this way of life into mere simulacrum. Anyone advocating for anything other than a blanket rejection of generative AI in the humanities should be ashamed of themselves for being complicit in ushering in a new dark age.
Well said, Alexei!
You nailed it….thank you
Setting aside the unnecessarily harsh and sanctimonious tone of your comment, why don’t we let a thousand flowers bloom? Jimmy is not prescribing any way of doing philosophy, simply describing his own. No one is forced to use AI. Some people write while high or drunk, others take inspiration from deities or from psychedelics, yet others receive feedback from dozens of peers at elite institutions. Lucretius wrote in verse, Nietzsche in aphorisms, Socrates never wrote. Older faculty still use double spaces and underline book titles and rely on RAs for the bulk of their literature reviews. Younger scholars use many cool widgets. Others pay for the professional services of editors. Most of us use all kinds of tech in all sorts of ways. Let people experiment and judge the results.
Unnecessarily harsh? I respectfully disagree. We are currently witnessing the slow annihilation of philosophy and of humanism more generally–not just at the level of established academics using these technologies, but of the conditions for the social reproduction of philosophy as a practice for the next generation. Students are coming into undergraduate programs quite literally without a high school education (going off the standards of just ten years ago). Some of them are functionally illiterate, and that is not an exaggeration. People who have had access to generative AI for the entirety of their undergraduate educations are now coming into graduate programs and are quite literally incapable of writing. Give it another six years, and we’ll be seeing our first crop of junior academics publishing their AI-slop journal articles with institutional affiliations that they acquired with their AI-slop CVs. They’ll be submitting AI-slop dissertation reports, teaching classes with AI-slop slides and AI-slop lecture notes… I genuinely cannot wrap my head around the fact that this is happening and so many of us seem to be so mind-bogglingly cool with the enshittification of the university that a bunch of weird nerds in Silicon Valley have thrust onto us. But everyone from the big kahuna admin to the precariously-employed adjuncts have agreed, more or less across the board, to let our standards go down the toilet instead of alarming society by continuing to hold students to the same standards they were held to ten or fifteen years ago and watching as dropout rates suddenly skyrocket. You cannot dissociate the pedagogical armageddon that this technology has wrought upon us from its usage by “professionals”. So yes, I will continue to be “sanctimonious” in the face of unconscionable philistinery emanating from within our own walls.
Your equation of generative AI with the other practices you identified is simply disengenous–as though a dram of scotch or a bong hit can be compared to literally outsourcing thought to a black box trained on Reddit threads that generates hallucinations based on a sophisticated version of text prediction which, again, sounds like an absolute nightmare to anyone who actually enjoys the process of thinking and writing. The better analogy in this case would be like if an athlete got a droid to sub in for him on the field, or a chef used ChatGPT to help design the menu at their restaurant, or a painter got a robotic arm to touch up their canvas. Why would anyone who actually finds this way of life and its associated activities pleasurable want to outsource it to some third party? Unless, of course, all you really desire is the external goods of academic life, and you see the actual practice of philosophy as mere dross that you need to slog through to get to what really matters: a cushy six-figure salary, lifetime job security, and tons of praise and attention from people who call you “professor”.
I’m all for letting a thousand flowers bloom. But in this case, the “flower” in question is a weed–and only the lowest of men would look upon this weed as a rose.
No clankers were consulted in the drafting of this reply.
Much appreciation for how well you write, and for saying the things you say. To me everything you say seems right on target. Be careful though. Dopamine hits from getting “likes” is part of the story of our enslavement. It’s useful for clarity to have proposals like Licon’s on the table for everyone to consider and he is generous to provide it.
I said “everything” but actually not quite everything. cf the comment below about the capabilities of frontier models.
Much to chew on in this very eloquent statement. I’ll just respond to this point about ‘equating’. I did not equate these other practices with the use of AI; I listed them as some of the many ways people write. Enhancing your writing with AI like Jimmy does is one other (not mutually exclusive) way. But you write:
“literally outsourcing thought to a black box trained on Reddit threads that generates hallucinations based on a sophisticated version of text prediction which, again, sounds like an absolute nightmare to anyone who actually enjoys the process of thinking and writing.”
I don’t think what Jimmy does should be seen as “literally outsourcing [his] thought”. Nor is this an apt characterization of the capabilities of current frontier models.
Trust me, I do enjoy thinking and writing. Your description does sound a bit nightmarish, but the good news is, it’s just this: a nightmare, not reality. But then maybe I’m just one of the lowest of men, appreciating the humble weed while the aristocrats tend to their roses in their shrinking garden.
PS: I don’t think the functional illiteracy of high school students has much to do with AI. It’s part of a long term decline that started long before LLM use spread and which the pandemic response likely accelerated. Maybe LLMs will accelerate the trend even further—I worry about this with my own students, but the claim requires evidence and does not bear on what Jimmy is doing.
I appreciate the magnanimity of your response, and perhaps only want to highlight that a core fear of mine is that in about 40 years, conversations like this (in terms of wit, eloquence, insight, and the rest) will be structurally incapable of occurring because our societies will have stopped producing witty, eloquent, and insightful individuals as a result of everyone using generative AI as a crutch. I agree that this general trend began earlier than ChatGPT, but it has certainly acted as an accelerator of an unconscionable order.
That said, I firmly reject the language of “enhancement” in the context of using ChatGPT as a tool in the writing process; I can hardly conceive of it as anything other than a debasement. We sigh when we encounter AI slop writing from students, yet we want the exact same LLM that produced it to touch up our prose? Why? Because we don’t trust our own judgment regarding the quality of our prose to such an extent that we want to defer to (or even just consult) the piece of software that generated the AI slop you had to mark a few weeks ago?
But that’s just me. I see form and content as inextricably bound up with one another. I realize it’s apparently a somewhat hippy-dippy view to hold in academic philosophy. But it’s not lost on me that it is exclusively the people who I know to be awful writers that have hopped on the AI wagon, and that the people that I know to be particularly good writers are the most outspoken and passionate critics of AI.
It’s interesting to note that conversations like this were impossible 40 years ago as well, in terms of who was able to participate (both contributing and reading). Online communication has degraded some of our communication abilities and enhanced others. I expect the introduction of AI to accelerate some of the changes we’ve already gone through, undo others, and move in quite different directions as well.
Just chiming in here to say that there is a very good case to be made that the post-2010 decline in literacy is almost entirely due to the viral spread of digital technology, both inside and outside the K-12 classroom. The larger philosophical point that Alexi is making isn’t affected by the fact that AI isn’t directly responsible here. The basic process is the same, we develop a digital “tool” that outsources and distracts, and are then surprised when students are less capable and less focused. Our institutions are basically set to auto-adopt any shiny digital toy and this is arguably reversing the largest educational success story in the history of humanity (the spread of mass literacy in the 20th century).
I completely agree about the root cause — screens, more specifically phones and social media — but there are multiple causal factors at play that are worth teasing apart, and the evidence suggests that much of the illiteracy we’re observing is the compound effect of factors largely independent of AI adoption. This is not to say AI will not worsen the trend—I actually think it will if we’re not careful. My only claim was that our current observations are at best weak evidence for that. It’s also not obvious to me that AI and social media impair learning in quite the same ways or the same direction.
Very well said. This shit is enraging.
You could tell, really? Are you some kind of detective? You must be. Congrats to you for cracking the case.
In case any readers have an article’s worth of thoughts on the topic of whether AI can/should be incorporated into ethics research, my colleague, Philip Nickel, and I currently have an open call for a Topical Collection on this question in Philosophy & Technology: https://link.springer.com/collections/hgbbichffg
Some of the above comments worry about cognitive atrophy and about losing our human capcities which are important for philosophy. But that’s an odd critique because Licon makes abundantly clear that he doesn’t think AI can replace those skills, and he makes clear that he uses AI only at certain steps of the process so as to preserve those skills. I worry that some of these comments reflect a wider view that *any* use of AI is corrosive to thinking; and that view seems patently false, just as much as the claim that computers must corrode our thinking. Some patterns of use of AI (and any number of other technologies) do reduce our capacity for the kinds of thought we care about, but not all.
Licon makes clear that he uses AI *as a discerning expert*. The reason I ban my students from using AI is that they’re not experts, so I don’t think they’ll be able to use AI effectively and avoid mistakes.
There are plenty of possible criticisms of Licon’s approach to AI, of course. We might worry about environmental impact, privacy, and beyond.
Isn’t the worry about cognitive atrophy and losing capacity less a concern about what happens in a particular individual (an adult, let’s say, with fully developed brain and expertise in the area of in which it is proposed an LLM be applied), and more about what happens in the society at large into the next couple of generations?
The suggestion would be that certain types of cognitive offloading will result in the educational/social/economic/political infrastructures for developing human experts to be weakened. This would include the infrastructures for developing engineers or medical experts, not only philosophy or creative thinking in other areas. The atrophy would seem also to be one that can be expected to accelerate.
Thanks for this response, Runa! Yeah, I straightforwardly agree with you that that’s a big concern. That’s somewhat different from what I understood some of the comments above to be saying, but if that’s what folks meant then I withdraw my point.
And, as I said above, I don’t want my students using AI because I’m afraid that AI tools will be used in ways that are harmful and for which they’re ill-suited.
In the foreseeable future, LLMs will produce publishable papers from an iterative back and forth of prompts and responses. You won’t actually have to write anything (other than the prompts). If I’m right, what value will publishing philosophy papers have? Just proving that you have the chops to do it?
What value does it have now? Just proving that you have the chops to do it?
The value that people derive from reading them? Isn’t this why journals exist?
Right, the question is why a human would put in the time and effort to write the paper when an AI could do it. It’s a question about the value of our doing so.
At least for now LLMs don’t sit around writing stuff on their own, but if and when they do, then people may have to practice their philosophical chops in new ways. If you find value in writing then write.
Happily, we won’t have to read it either! We can just run prompts that give output responses in our stead while we do nothing but consume and vegetate.
Remember, there’s what people say they do, and there’s what people actually do.
“What matters, in the end, is that the argument I sign my name to is one I understand, endorse, and can defend without any of those tools in the room.”
Perhaps I’m taking the above too literally, but if this is the standard, then what’s wrong with plagiarism?
Plagiarism denies credit to someone who is due credit. I don’t see that that applies here.
Interesting! I’d have said that plagiarism claims credit where it’s not due. Ordinarily, of course, these two things go together, but AI seems to break them apart somehow.
I agree. Ironically, sometimes I generate ideas that I first like and then vehemently oppose to. So I can fail to endorse stuff that I authored!
I’m writing my second book, and this time I’m using AI more frequently. For one thing, I’ve been through a lot of trauma, and have cognitive challenges. It would be no different than a man going to a grocery store needing a cane to make it there. That man would have physical issues, I have cognitive. So AI helps a lot! My creative mind is intact beautifully, but I sometimes get overwhelmed cognitively. I’ve had this disability since I was traumatized as a very little boy. AI helps me in many ways, but it is still my writing in the sense that the director would direct actors. The movie turns out the way the director anticipates or wants, the actors are simply props making it happen. For me, AI is support or props that enhance my writing, but absolutely does not create it as far as theme, story, but does help in structure, and other ways as far as writing techniques. I use it a lot to check my writing. I asked it if this sounds good grammatically, but I also use grammarly for that. The bottom line for me at least is it I love to write, be creative, change People’s perspective through writing, and AI basically makes all that happen many times faster. I will be able to write more books because of it. But the stories will be my own, just like the movie will turn out exactly like the director intends. I think we should have a class of writers that are pure writers, educated, well versed on every aspect of language, but for people like me, who love to write mostly to try to help people see things differently, writing is a vessel for more people than just those who went to college and learned how to. I think it’ll open up the whole system so that more people are able to contribute in ways that make it a better industry altogether. But I do think we should have distinctions. People like me, should be readily open about it, where’s someone who is a professional and does not need AI necessarily, should be looked at as a professional writer in the pure sense. Anyway, this has been weighing on me which is why I gave such a big response.
The amount of whiny irrelevant cope in this comment thread is pretty amusing. Good luck with that.
Just when I was about to say that people were being needless harsh…
I found the description of your workflow interesting, thanks for sharing. I disagree with the attacks and denigrating comments, but some commenters raised interesting questions and objections, which I’d like to see you address with more grace.
One thing I’d be curious about is if you could give a concrete example of a piece of work where you’ve found AI substantively helpful—e.g., at generating an objection you had not anticipated or coming up with an original conceptual distinction. It might go someway toward persuading some skeptics if you could show (1) that your prompting was key to unlocking the insight you got from the LLM and (2) the insight was one you would not plausibly have had on your own. This would demonstrate (to me) a genuine kind of collaboration.
And to the folks who object to my AI approach, can you actually explain how my approach differs from relying on a research assistant in the same ways? Is there a difference, really? C’mon. Get real.
I don’t necessarily object to your approach, esp. on a quick read. Not all uses of AI in research and writing are inappropriate.
But since you asked about the difference between your approach and relying on a research assistant:
I assume you would acknowledge a human research assistant in your papers and presentations, as you should. Do you also acknowledge AI use in your papers and talks?
I didn’t see that you discussed this, again on a quick read, and maybe you’re already doing this. Such acknowledgements can go a long ways toward responsible AI use, so much that not having them is a big problem…
You would let an RA touch up your prose?
I haven’t read your whole article and am not here judging its content, just replying to this one particular request. Relying on a research assistant is helpful to the research assistant in developing the kinds of skills people are referring to elsewhere in the thread. Relying on chatgpt for it instead contributes to a loss of that particular way of helping train people up and helping people practice and hone academic skills, which may be bad for the profession and for human development long term. I am not here proclaiming a final judgment. I can imagine other means of training/practice replacing this, that are less potentially exploitive, maybe.
This was (and still is) a big concern 15 years ago when AI journalists emerged on the scene, e.g., Narrative Science.
That AI was very different from LLMs today; it was more of a paraphraser and didn’t generate new content like ChatGPT. But it was still great at sifting through, say, long and boring quarterly financial reports to sniff out the highlights and write a news article. And AI was often tasked with local sports reporting, e.g., high-school sports, which aren’t the most interesting things for humans to cover, esp. for newsrooms that were short-staffed anyway.
But however boring these stories might be, it was still important experience for cub reporters to have, who are new to the profession. If they can’t cut their teeth on these sorts of stories, where are they going to get experience for more important, investigative reporting later? Without new reporters gaining experience and growing into senior reporters, journalism as a profession would be in serious trouble (along with a democracy that relies on a free press).
Similar concerns exist with LLMs and researchers. Helping to train people and give them practice might be an annoying time-suck for some people, but that’s what’s needed if you want to raise competent people and experts in the future.
Otherwise, we’d have little choice in the future but turn to AI for journalism, research, coding, whatever. And that’s exactly what certain tech bros are trying to sell you.
I got a BS CS, 9 years professional software design experience, a suite of patented algorithms. I’ve supervised teams of many dozens of coders.
Most coding is barely more than stupidly grinding away at trivial yet time-consuming tasks. It’s horrible clerical grunt-work, and, as far as I can tell, most software engineers are happy to be relieved of that grunt-work, and they’re very happy to be using AI.
Eric, you have to start zooming out and asking bigger-picture questions. Individually, people are almost always happy to be relieved of some particular labor. But they will often feel very differently if they vividly imagine a world where all labor of that type is outsourced, if they think carefully about the huge amount of human sociality and human development that depends on the labor being performed by humans. Most coders, I assume, do not want human coding itself to end, because they’ve built their lives on that career and that skill set (perhaps you’d call them Luddites here?).
There is a line that we cross, where we stop just relieving drudgery and start replacing vitally important and meaningful labor, such as the work that an actual paid human RA can do for their professor. Crossing that line en masse literally means the end of human society as we know it. That’s the concern.
I’m not sure what you’re claiming here.
AI automation is going to end some types of work, it’s going to produce some new types of work, and just generally make both positive and negative changes.
I was an RA for a spell while I was in grad school; I wouldn’t call it “vitally important and meaningful labor”. It was stupid grunt-work. And nobody wants to be a coder. It’s degrading factory labor.
Yes, there are huge dangers associated with AI automation. I’ve repeatedly said it’s an existential threat to philosophy, unless philosophy changes.
But “the end of human society as we know it”? Yeah, I grew up in the 1960s, before cell phones, before computers, before internet, before all of it. That society ended entirely. Good riddance. What comes next, we don’t know.
If he’d said he used a model trained independently instead of a corporate product, he might have a valid point, but as soon as you introduce commercialised applications there is no ethical use case. These products are ethically compromised at their very core. It’s like saying you’re somehow able to wield the evil thing built by evil, from evil itself, designed specifically to do nothing other than evil, the right way.
I haven’t seen the Luddites giving any arguments here, just making claims about things like AI and cognitive atrophy. No doubt excessive AI use can cause cognitive atrophy. But the OP is advocating specific AI use-cases that may well build cognitive skills.
There isn’t a much cognitive skill involved in doing literature searches. Using AI for literature searching is just a time-saving tool. It’s no different (as others in this thread have pointed out) from using Google Scholar (which is also AI powered). I use ChatGPT for this a lot (e.g., what happened to substantial forms after Descartes?). It’s caused me to enormously broaden my literature searches. And I’m now much more eager to do literature searches. Increased practice is making my skills better. So I think AI increases skills here.
Using AI to assess the state-of-the-art in another discipline is also positive use-case which builds skills. Today I wanted to know how much of Xiao-Gang Wen’s qubit ocean program is legit science and how much is speculation. It would probably be impossible for me to find a physicist who could answer this. So I asked GPT-5, and I got an excellent run-down on which parts are established science, which parts are pretty secure but not fully settled, and which parts are pure speculation. I asked because I’ve read a half-dozen or so of Wen’s papers, but I just don’t know enough physics to know when he’s shifting into speculation. And I know enough physics to follow GPT-5’s explanations. Having an on-call research assistant with pretty high-level expertise in almost every field is an immense gift of AI to all academics. Without it, I would have made mistakes. Seeing Wen’s program from the perspective of a physicist helped me learn a great deal about that program. Plus I learned that condensed-matter physicists have some sharp disagreements with particle physicists. Here using the AI is helping me ask lots of much more highly complex research questions. So this use-case builds lots of cognitive skills.
Using AI for brainstorming is another case where it’s hard to argue for cognitive atrophy. It’s analogous to having a discussion with a well-informed human, which I don’t think induces cognitive atrophy. A great benefit is that the AI can talk about lots of subjects. So using AI might increase cognitive strengths here, as you can now really pursue very deep conversations without suddenly running into human limits. (Perhaps folks at really elite institutions have quick access to lots of experts. Most of us don’t, so I see the AI as a positive tool for democratization of expertise.)
And you can engage the AI in weird philosophical topics without having to deal with the incredulous stare. Moreover, you can use the AI as a sounding board, and start to see whether your idea has any depth, and what it’s connections are to other topics. Again, I think this has enabled me to think about research ideas that I never would have been able to think about, because I just don’t have the breadth of knowledge possessed by the AI. So I’ll put this down as a positive cognitive skill-building benefit of using AI.
Most of us lack proofreading skills – we just don’t have them. We miss lots of errors in our own typescripts, which are now often thankfully corrected by AI embedded in word processors (not very powerful AI, but it’s still AI). No skill is lost here. And this is parallel with the case of conceptual proofreading (which is what the OP is talking about when he talks about using AI to find weaknesses in his reasoning). No skill is lost here. On the contrary, skill might be gained – if you can get lots of critical feedback on your reasoning and writing, you’ll probably become a better reasoner and writer.
When it comes to the sorts of use-cases in the OP, AI can help build lots of skills. (Am I an AI? No. I’m Eric Steinhart, and I sign my name to every post I make here.)
Maybe folks already know about this study. Not exactly cognitive “atrophy” but maybe we could say cognitive or at least attentional degradation.
https://arxiv.org/abs/2506.08872
The use-cases in the OP do not include essay writing.
By the way, I’ve been wondering. What’s “a use-case”? I’ve noticed the administrative staff promoting the use of AI in courses at my institution using this term. Does that word mean anything other than “a use”?
I think the distinction between a “use case” and a use is like the distinction between a “teachable moment” and actual teaching.
Good question. It’s basically just a way that a tool can be used, but usually one that’s spelled out with some precision, and that is something the tool was designed to do. So a tool-maker might write a how-to manual about a use-case. But just incidently using a hammer as a paperweight probly isn’t a use-case.
I would want to be careful with the “brainstorming” sort of case. One of the important techniques in a group brainstorming session is to have each person do some solo brainstorming before anyone hears anyone else’s ideas, because the first few ideas that are stated tend to channel other minds into following them. If everyone’s using the same few AI tools from the start of their brainstorming sessions, there’s going to be an indirect homogenization (the same way as when everyone is citing the same few early papers on a topic).
It’s of course possible to break out of this, and AIs are usually more willing to “yes, and…” a weird initial idea that a human gives, while other humans are more willing to shut it down.
But brainstorming is definitely something you want to do un-aided for a lot of the time as well.
Yeah, all good points. When I’ve used GPT for brainstorming, the conversations are usually pretty adversarial, where I really have to wrestle with the AI to get it into the weird space my own head occupies (“but what if the monkeys had screwdrivers?”). That wrestling itself can be very productive for me to clarify what I’m after. But if you’re not willing to wrestle with the robot, there’s a danger of homogenization, or just shallowness.
I agree Eric. I am afraid the situation here is very much analogous to a scene from the movie “Money Ball” – for those not familiar, a group of old fart baseball scouts are sitting around a table insisting that Brad Pitt’s character Billy Beane is wrong to be using these newfangled computers and statistics as a way to scout baseball talent. They were saying that the old school method of just looking at the players with your eyes and going by your gut is the best method – rather than to let the computers crunch the numbers and spit out a ‘wins above replacement’ metric. Billy Beane fired them all and said you’ve got to “Adapt or die.” Now his high tech algorithm approach to evaluating baseball players is what everyone uses, and the old farts and their old ways have been put out to pasture. The point being: When your environment changes technologically, informationally, or institutionally—clinging to inherited methods can be fatal. Rather than going full ‘old man yells at cloud’ about genAI, we need to adapt, and the last to adapt will be … like those old timey baseball scouts, out of a job.
Is this really an analogy to Moneyball?
In baseball, stats are used to predict the potential of players. AI (LLMs) isn’t used like that in philosophy but to generate ideas, which is a very different kind of animal.
Maybe you’re suggesting that philosophers pay attention to data, which may be fair…and already happening in many areas of philosophy.
Stats and number-crunching are well established methods, even 25+ years ago in Billy Beane’s days. LLMs are not, and even its designers can’t tell you exactly how they work. And the more you know about LLMs, the less reason you have to trust it. E.g., they’re not hallucinating just some of the time but all of the time; it’s just that some hallucinations are useful and track reality.
Your broader lesson seems plausible, though: “When your environment changes technologically, informationally, or institutionally—clinging to inherited methods can be fatal.”
But given that LLMs are still a new phenomenon that may or may not last, we don’t know with much confidence if our environment is really “changing” as opposed to being currently challenged. We certainly don’t know with much confidence if leaning into LLMs is the right or only evolutionary adaptation.
And the “inherited methods” of doing one’s own reading and writing have been proven for thousands of years. In contrast, using vibes in baseball player analysis has never been reliable.
All the general points you’re making here are right and important ones!
I do want to pick one nit though:
You say “the more you know about LLMs, the less reason you have to trust it. E.g., they’re not hallucinating just some of the time but all of the time; it’s just that some hallucinations are useful and track reality.”
It’s notable that at least some people have been saying that human perception is also just “controlled hallucination” since at least 2019: https://www.edge.org/conversation/andy_clark-perception-as-controlled-hallucination
Some people have taken these sorts of discoveries about human perception to be reasons not to trust human perception, the way you’re taking them to be reasons not to trust LLMs. I would want to keep the “healthy” in “healthy skepticism”.
We definitely don’t have a clear sense of what LLMs are good for yet. But I also don’t think there’s any reason to say they’re good for *nothing* in philosophical research.
Quick reply for now, Kenny:
“Hallucinations” is a contested term in convos about LLMs. Other folks have suggested “fabrications”, for instance, to not inadvertently imply the existence of an AI mind.
So, I think we’re talking about two different kinds of hallucinations. Mine isn’t literal but only about generating info or claims from a body of data, as opposed to retrieving info as search engines do.
In generating that info, LLMs are generally not grounding their claims in things that are held up to be true. That’s what I mean by “hallucination.”
At their core, LLMs are simply predicting what the next word, in a string of words, should look like for the entire text to sound plausible to a human.
And that’s it. All they do is to make plausible sounding conversations. As their name suggests, they model human language, as opposed to modeling truth.
No, it’s not correct to say that LLMs are just “predicting what the next word . . . should look like in a string of words for the entire text to sound plausible to a human”.
There’s a by now pretty large literature on the internal representational structures of LLMs, and those structures are very complex. LLMs have world models, including models of space, time, numbers, etc.
LLMs are probably doing some version of predictive processing much like (but not entirely like) the predictive processing done by human brains.
Recently in a discussion with a student, I was told that LLMs have internal models and hypotheses and generating programs of human’s world view, but not of the world itself. The latter seems to be what all the current efforts are directed at. Smart people have been recruited with 1 billion paychecks for this purpose. And in the discussion, they were trying to convince me to contribute to it.. (No value judgment, just reporting something I learned)
Look at the LLM models of arithmetic, which are helical, and probably far from anything a human uses (we seem to have a neural number line).
Are you talking about the Kantamneni and Tegmark paper? I thought that was a first stab at explaining their ability to do arithmetic – some evidence but not fully established. What other papers you would recommend?
Hi Eric, since you seem to believe LLMs more than me and don’t mind reading its work, here’s what GPT-5.1 says about it:
Here’s the prompt and details if you want to inspect them.
As I had said, my comment was a quick reply as I was heading out the door, not a careful technical discussion.
But the idea is basically right. Feel free to argue with an AI if you like, but I don’t have time to be drawn into the convo…
I have a suite of patented AI algorithms. I study real computer science.
I’ll do the first one for you, building upon the previous chat above.
But please direct future replies about this to GPT-5.1.
Ah, vested interest would explain a lot here.
It cuts both ways, doesn’t it? You have vested interests too. I wrote about this here, if I may engage in a little bit of shameless plugging.
https://open.substack.com/pub/nicolasdelon/p/publishing-without-romance?r=di5yu&utm_medium=ios&shareImageVariant=overlay
Vested interest explains what? I was a computer scientist (still am). I worked on early AI back in the 1990s. So what.
Are you saying I’m biased in favor of AI because I actually understand it? (As opposed to those here who clearly don’t understand it.). That would be accuracy, not bias.
Or are you saying that I’m biased because I make money from AI? I don’t make any money from my patents (they’re owned by a company, all I got was my salary long ago). But so what again. Anybody here who’s anti-AI could easily short AI stocks, and thus have their own “vested interest”.
If you’ve got an argument against my philosophical positions, then make it. Otherwise, knock it off with the ad hominems.
By the way, I sign my name, I don’t hide behind anonymity.
A common argument on Twitter: “Only us who own and mint crypto understand the technology. Obviously anyone who doesn’t engage with it is a fool who just doesn’t get it.” These days more commonly made for AI.
Didn’t expect to find a variant of that here. That someone who understands and thinks a technology sucks would refuse to engage with it never occurs to its evangelists.
You’re right about this. Many people who understand the technology much better than any of us do think it “sucks” for one reason or another. Some advocates don’t really know much about it, and many critics do know a lot.
However, I think Eric was just taking issue with your accusation that his vested interest explained why he would hold the views he does, and that’s uncharitable, as another commenter would put it.
None of this bears on the question at hand, though: whether LLMs are just next-token predictors (I personally think this technically correct point undersells what the technology does, but we can have reasonable disagreements about what this fundamental feature does or does not entail.)
As respectfully as I can, it’s not really an open question whether LLMs are just next-token predictors. They simply are.
If anyone doesn’t believe me or ChatGPT above, then maybe Sir Demis Hassabis (co-founder of DeepMind) is credible here. See the last line from an article this week:
Demis is only the latest of high-profile AI leaders to say that LLMs are a dead-end.
On the need for a world model, Gary Marcus has always been prescient about AI; see his article from last summer.
A world model would give AI the grounding it needs to avoid “hallucinations” as understood as something like presenting a claim without a regard to whether it’s true or false. Alignment with an accurate world model would be a reason to think the claim is true, i.e., a reason to trust.
In other words, at their core, LLMs today have no intrinsic connection to truth (but could have connections if part of a larger system that double-checks LLM claims, e.g., RAG).
Again, LLMs don’t care about the truth or falsity of their claims, just whether the next word they predict, in a string of words, sounds plausible to humans.
(Yes, there can be encoded logic or knowledge in the textual patterns that the AI learns, as Luciano Floridi et al. explain. But this isn’t a direct or intrinsic connection to that knowledge, just a function of statistical frequency.)
But I get the desire for some folks to want LLMs to be more than this. LLMs can seem surprisingly capable despite this untethering to the ground truth, but that competency is just the A illusion. Like Magic 8-Balls, they can sometimes seem wise and all-knowing, until we understand how they work.
Have a nice weekend, all.
I believe they are next-token predictors! But saying they are undersells what they do, is all I said. Sorry if that wasn’t clear. I’m not disagreeing about this.
Yes, you were clear on that, Nicolas. I wasn’t directing those comments to anyone in particular.
Sure, any one-sentence description of LLMs (or anything else) can undersell that they do. I’ve long said LLMs can have some utility, though it can be coincidental (because LLMs are untethered to the truth) and so accuracy must always be verified.
But that one-sentence description — the same one that Demis and others have used — captures the essence of what LLMs are: they aren’t designed to deliver truth but just the next token in a sentence. It’s a great feat of engineering, but not of epistemology.
And that’s a big reason not to trust them as anything beyond fun conversational partners. They’re hallucinating all of the time, in the sense that there’s little or no assurance that any given LLM claim is true or not, yet the claim is still presented confidently as true, insightful, whatever (and sometimes it coincidentally is).
Overestimating LLMs is a root cause for disagreements in these discussions, and that seems much more prevalent than underestimating LLMs. So, reducing LLMs to their core function, in the one-sentence description, helps to remind us that they’re not magically knowledgeable or even trustworthy if accuracy is important at all. They just model human language.
If you agree with that, then we’re not disagreeing on any big, maybe just around the margins.
Anyway, thanks for the convo here, even if it’s been tense at times. I think we can all move on for now, or at least I hope to. Enough’s been said, and I’m sure we haven’t seen the last post here about AI!
Thanks, Patrick.
I guess I’m a bit of a deflationist about truth so I don’t get hung up on LLMs’ deep relationship to truth (or lack thereof). It seems like part of the apparent disagreement lies in whether one sees “just next-token predictors” as a rhetorical sleight of hand or a substantive claim about the fundamental nature of LLMs (if not deep learning more generally). “Just” tends to be load-bearing in these arguments, but it glosses over a lot, both about what’s going on under the hood (transformer architecture and all that), and about what it can achieve in the real world. After all, no user is interested in predicting the next token; they are interested in generating text, and as Zvi puts it, frontier models do provide a lot of “mundane utility.”
To me, the idea that they are “just next-token predictors” is kind of analogous to the truth that the brain is just a meat machine. Technically true, profound, humbling, and an important reminder. But it undersells a lot. If I may steal from Dennett, we might be dealing with real patterns. There’s a true level of description that can bring the patterns out of focus, but they’re nonetheless real.
Agreed, but if economists are right, margins are where the action’s at.
To you and Patrick, what exactly is meant by a “next-token predictor”?
It seems trivial: any discrete process is a next-token predictor.
Or, for precision, what does “predict” mean here? That it’s running a Markov model? Extrapolating curve-fitting? Tracing geodesics in some extremely high-dimensional probability landscape? What?
Or consider that LLMs are likely to model probability distributions over their entire context windows (which run into the 100k or so of tokens). What’s the “next token”?
And what’s the contrast? One of our very best theories of cognition, namely, predictive processing, is pretty much a “next-token predictor”. On that view, LLMs and humans are equivalent.
I’ll say nobody knows how LLMs work. The math involved is very, very hard.
I’d agree with you, Eric, that “nobody knows how LLMs work” exactly.
And that’s a huge reason to not trust AI, even if we’re able to nail down a definition of “next-token predictor.” (Maybe it involves POMDPs..I dunno…)
I would reject your equivalency of LLMs and humans, though. Or at least it’s much too premature.
For one thing, as I linked to above with Gary Marcus’ article: humans are typically grounded in a world model. If our claims don’t align with the many other things we’ve accepted as true about the world, that’s an important safeguard against making false claims.
LLMs have no such safeguards. We can add RAG etc. to help improve accuracy, but this isn’t how LLMs are most commonly used (e.g., by students, etc.).
So, unlike humans, LLMs don’t take the extra, crucial step of evaluating an entire claim against a world model (because that tech doesn’t yet exist).
Another way to think about it: LLMs try to model our language (as a proxy since they don’t have direct access to the world), and our language tries to model the world; so, LLMs face a serious GIGO problem that we don’t in making claims about the world. Our access to the world is more direct, and we don’t need to rely on an unreliable proxy (human claims) in jumping that wide chasm.
Maybe someday AI will have direct access to a world model, so you could be right that AI and humans are more alike than we may think now, even if today’s AI isn’t comparable.
But that’s also something no one can say with much justified confidence today, esp. as the human mind is very much less understood than LLMs (and we agreed that no one knows how LLMs work).
So, equivalence between AI and humans is just a bare logical possibility today. That doesn’t seem enough to hang anything on, such as trust. Meanwhile, there are good reasons not to trust AI today.
P.S. I’m not trying to explain AI to you — I trust that you have the technical background to already have a good idea. But there are different ways to conceptualize what the forest looks like, and if you’re working close to the trees, it might be hard to appreciate those other views…
And I hope that you know that I’m taking your (and other) comments seriously. That’s why I’m investing so much time in these responses, despite not having much time.
Gotta go — thanks for the convo, and have a nice weekend.
Yes, I’ll agree that trust is a serious problem here. AI is very powerful and very dangerous. I’m more inclined to worry that it’s getting close to super-human in many areas. It becomes uncanny and godlike. (But it’s not going away, so I think we’d better figure out how to deal with it in positive ways.)
This is such a weird post and such an aggressive comment section. Peace out.
Interesting! I wonder how much having the synthesis written by AI shapes how you construct the first draft. When my students use AI ‘just for structuring’, the result is a very similar set of papers, where all of the beats/moves are in the same place, even if the language is different. I’d be worried that AI here is short-circuiting some of your creativity — or worse, failing to give credit to someone else’s idea that it scraped for your response.
Yipee. More shilling for AI from Daily Nous. It’s funny how society at large seems to be realizing AI ain’t all it’s cracked up to be faster than some philosophers, though the comments here give me some hope.
Like one other comment said, what a time to be on the job market, putting to pen the best work you can to compete with hundrends of other able candidates, while (some) senior academics safely enconsed in their cushy jobs argue in favour of increased laziness. Was at a faculty meeting recently where the tenured faculty argued that we should just let students use AI because it’s too much work to find them out, and all jobs will be using it for everything soon (even corporate stiffs don’t buy that anymore). Boggles the mind.
Clearly you haven’t used ChatGPT Pro, the one that costs 200 a month – it is , literally, beyond what it’s cracked up to be.
It’s certainly not “all it’s cracked up to be”. But too many people go equally wrong in the opposite direction by denying that it has *any* value.
Agree with you here, Kenny. Both sides would be denying the obvious.
But I suspect that even those who deny that AI has any value are adopting that position as merely a rhetorical strategy:
Besides not wanting to give their opponents an inch, they may be concerned that acknowledging any value may give the wrong impression of endorsing the use of AI, even in limited use-cases.
That is, they may believe there’s such an overwhelming list of reasons not to use AI, that even saying “it’s ok to use AI as long as it’s done responsibly” would be taken as general permission and enable more misdeeds.
Even if some people do use AI responsibly, it seems the vast majority do not. Even some of the folks commenting here in Daily Nous, a professional community, seem ok ignoring the basic norm to be transparent with AI use in writing, etc…including in discussions about the responsible use of AI, no less…
Or maybe some AI value-deniers just haven’t played around with AI. But I think it’s a grave mistake to not “know your enemy.”
Anyway, I’d think that rhetorical strategy is more likely to backfire once outside your echo chamber and faced with people who have seen, with their own eyes and ears, what AI can do…
I think it’s the senior faculty with cushy jobs who have the strongest incentive to argue *against* AI, precisely because it threatens the status quo and what they’ve spent years investing in. I’d wager adoption is greater among younger folks, precisely because of the competition—and you can interpret some of the reluctance to adopt as a desire to reduce competition.
Interesting hypothesis, even if uncharitable, but is it supported by anything?
Most of the anti-AI folks I know seem to be newer scholars, who also tend to be more the activist type than us in the older set. As Huey Newton (co-founder of the Black Panther Party) had put it, “The revolution has always been in the hands of the young. The young always inherit the revolution.”
Sure, the senior faculty may naturally get the most attention. But that doesn’t mean there’s not a groundswell of younger people alongside them, including students and non-academics who have no stake in “reducing competition” (which, btw, is a weird thing to accuse academics of, esp. if you can’t point to any examples of this behavior ever happening.)
So, an alternative, more charitable view is to simply take those critics at face-value: that they are anti-AI for their expressed reasons related to justice, equity, energy use, labor and IP exploitation, etc. They are anti-AI because they value the enterprise of academia and philosophy, and they have a hard time seeing how it can all co-exist with unchecked AI use.
No malice needed to explain their behavior. Is that really hard to believe?
PS: I linked to it above, but if you’re interested in my interesting, if uncharitable, hypothesis, I defend it in more detail over here (CW: no evidence).
PPS: I wrote a lengthy response that seems to be in limbo. But I’m very happy for you that you’ve never had to face competition for jobs, grants, or publications. Talk of a cushy situation!
Will try to give it a look, Nicolas. On a quick peek, I appreciate your very thorough AI disclosure! That’s refreshing to see.
As for a cushy situation: you might not know that I teach at a public teaching university. No PhD program, no grad-student helpers, heavy teaching loads with very little dedicated time for research (unless we find our own funding), etc. Anyone at an R1 school has a cushier job than I do. As for grants, the NSF (my primary mode of funding) has a single-digit funding rate, which doesn’t sound cushy…
I don’t mind human competition at all. Some might say I have an over-developed sense of competitiveness. But it’s AI competition that should be rightfully scorned, for reasons mentioned ad nauseum.
As something I haven’t talked about much, see my comment above about the Gish gallop. Since AI moves at digital speeds, it’s so easy to flood a marketplace of ideas with slop that everyone would need to wade through, before we can ever get to something worth engaging with (esp. if others, not you, fail to be transparent about their AI use).
Triage is very reasonable in this situation, and my red line is at not engaging with AI content. I can’t even keep up with all the stuff I want to read by my human colleagues—there would need to be a pretty damn good reason to dedicate even less time to them.
So, I’m worried that a competition with AI would be a war of attrition, not so much a war against other quality or worthwhile ideas. We can’t keep up at digital speeds, which is fine since humans have many other strengths.
P.S. I’m still open to the possibility that AI could someday make a breakthrough in philosophy, but that remains to be seen. And there are reasons to think it won’t happen anytime soon, e.g., if AI is just remixing or operating on existing content, i.e., interpolating and unable to extrapolate or think outside its box.
You missed the irony and the reference to the OP (“cushy jobs”). You wrote it was “weird” to “accuse” academics of wanting to reduce competition, as if academics were immune to competition. I don’t think it’s weird at all to think at least part of an academic’s behavior can be self-interested and takes place in a competitive setting. I’d love to live in a world where we don’t have to compete for jobs, publications, and so on, or a post-scarcity world where we don’t have to compete for resources, but that’s not the world we live in. It’s woefully naive to assume that academics are immune to the incentives that motivate every other human being.
Well, I mean if you’re self-reporting as someone who’d wish to keep certain people out of a field just to reduce your competition, then I guess I’ll take you at your word. (Accusations are often confessions, as they say.) So, fine, maybe some people have that incentive.
And it may even seem reasonable to say that it’s better for the individual if competition were lessened. It almost sounds like a truism. But I don’t think that applies here:
Academic research and knowledge generation is a collaborative enterprise, even if an individual may be presently focused on winning a grant, etc. We typically build on top of previous work. So, I don’t understand the desire to remove collaborators—i.e., rungs on the very ladder you’re standing on—from this broader enterprise, unless they were just unqualified or would do bad work.
And as I said previously, AI work will generally count as bad work, at least insofar as it can muddy the waters so much with slop that we’re unable to identify the stuff worth engaging in. Triage is reasonable here, maybe even necessary.
But let’s look at grant competition: sure, I’d love to win every grant I apply for. But I wouldn’t feel good about doing that by just declaring certain people to be ineligible. I’d have to have a good reason (i.e., beyond self-interest) to do this.
Connecting back to AI: as already mentioned, there are lots of good reasons why someone would believe AI-written proposals should be ineligible for funding.
More generally, we can’t assume people will always want to reduce competition. Think about sporting contests: would anyone really want to disqualify their opponent’s players so to more easily beat them, just out of self-interest and nothing else? Ok, again maybe someone believes this, but this seem far from the norm.
I don’t know any athletes who’d want to win by stacking the deck in their favor like this. Sportsmanship is real, and competition helps push you to be better.
Likewise, I don’t know any academic who would disadvantage a colleague or potential colleague just out of a selfish desire to reduce their own competition. Maybe we travel in different circles… 🤷
Therefore, this is a non-issue in my opinion, and that’s the last thing I’ll say about it.
But I’d be happy to hear your thoughts on what I said about the Gish gallop and triaging the firehose of AI slop to avoid a war of attrition. That seems to be much more important than speculating about motives, esp. without evidence or examples…
“if you’re self-reporting as someone who’d wish to keep certain people out of a field just to reduce your competition, then I guess I’ll take you at your word. (Accusations are often confessions, as they say.)”
“I don’t know any academic who would disadvantage a colleague or potential colleague just out of a selfish desire to reduce their own competition. Maybe we travel in different circles…”
What are you talking about? This is getting ridiculous. Asking for more jobs, more spots in journals, fewer PhD programs, and so on is a straightforward way of expressing your preference for less competition. The suggestion that academics would like it to be harder to publish or get jobs is ludicrous enough. But if now you’re accusing me personally of being some kind of sociopath, I’m done with you. Enjoy your sense of purity.
I’m sorry you feel that way—I wasn’t accusing you or anyone else of being a sociopath, but ok.
And I’m far from being pure; I just don’t have such a dim view of my colleagues and profession as you do, it seems.
If I were so pure, I’d just say, “No, LLMs can’t be used ethically, full stop. There’s a long and compelling list of reasons, even if some people think they can use it responsibly.”
Anyway, it’s pretty telling that you’re still choosing to ignore the hard issues and want to keep hammering away at the non-issues, like motive.
Further, it’s ironic for pro-AI folks——who typically urge us to “just evaluate the arguments”, regardless of who’s writing them and why (e.g., Polly Proudfoot’s comment a couple days ago)—to ignore the anti-AI arguments and instead smear the motives.
I don’t mind good-faith disagreements, but this is something else and not a productive use of time. If all ya got are ad hominems, then have a nice day.
I haven’t used a single ad hominem and have extensively discussed the issues here and elsewhere, but like I said, I’m done with you. I will not respond further.
Delon,
I don’t know if you realize it, but your interactions on this site routinely seem to devolve into these ugly skirmishes. Try taking a beat. Get some sun. Do something to chill out.
Hi Dee,
Thank you for your concern. Please call me Nicolas. If you prefer to be formal, ‘Delon’ is not proper.
I’m aware that this website attracts a lot of unpleasant people who will raise accusations of bad faith, impugn the motives of specific people (as evidenced here and no I didn’t do this), compare named colleagues with fascists while using the cloak of anonymity, and generally grandstand about their pet issues without engaging with the issues. Contrary to your belief, most of my contributions do not “devolve” in those ways, except when I call out bad behavior. But you are right, I often make the mistake of engaging with these people, which leads to these skirmishes. Oddly enough (or maybe I shouldn’t be surprised), there is relatively little effort to call out the bad behavior of some regular commenters (nearly all of them pseudonymous or anonymous), and there have been several bizarrely personal commentary about me, again by anonymous posters.
Call me old fashioned but I don’t take personal advice from anonymous randos on philosophy blogs. If you’d like to address the substance of my comments, go ahead. If you want to continue being personal, be a Mensch and identify yourself.
Nicolas
I don’t get where Nicholas is coming from. My impression is that younger scholars are much more anti-AI, and this is possibly related to the precarity that they are much more likely to have to face. What I mean is that they recognize how AI is built on theft of labor—including the labor of artists and scholars—and how this is intertwined with the conditions giving rise to the precarity that they already face. It would make sense then to be skeptical of a culture (not necessarily just the technology itself) that devalues their work, reduces it to “content” and them to “content producers” (and consumers), and encourages them to embrace a “career” of slinging generative slop out into the world, that will be “summarised” by other slop machines—and all for very little in the way of financial reward, much less job security. Why would younger scholars be positive about that?
Nicolas, no ‘h’.
Why would *senior* scholars be positive about that? Note I was only responding to the claim made about senior scholars in the parent comment. I’m not especially wedded to the idea that there’s a substantial generational gap. What we’re each observing is also filtered through layers of selection bias: of course you’re going to hear more criticism from young folks. But this has to be reconciled with the other thing we’re constantly told to worry about: rampant AI use among students (up to grad school admissions according to a recent post at Leiter). Are we to assume that the current young scholars are magically immune to the social forces that both senior scholars and young students are responding to? It may be, but you can see where I’m coming from now, maybe?
That’s exactly what I’m seeing, too, Felix.
Further, I might expect many senior scholars (but not all) to care less about their own future work because there’s less future in front of them. It’s the younger set that has more at stake and is feeling the pressure, not so much senior scholars who likely have more publishing and research options.
And I know many senior scholars who want to retire early (or already have retired) because it’s far above their pay-grade to deal with this AI cheating b.s. (which can feel like yelling at a cloud), in addition to all the other things we’re dealing with now. They’re tired and leaving the fight to the next generation.
All this weighs against a presumption that senior scholars have the strongest incentive to resist AI, even if we’re just limiting ourselves to self-interested reasons (as opposed to concern for the academy, etc.).
Anyway, it’s ultimately an empirical question, and motive doesn’t seem to have much or even any bearing on this AI ethics debate…
“… society at large seems to be realizing….”
Yes, it does seem to be. Even Jamie Dimon: “Rollout of AI may need to be slowed to ‘save society’, says JP Morgan boss” (Guardian)
He goes on to talk about society adjusting to massive job loss, and needing time to adjust to that. He doesn’t of course mention the other harms coming through our offloading onto AI much of the social and cognitive activities that make life worth living, nor the need for international regulations to protect against these things. Still, it’s a step in the right direction.
Yes, yes, those “massive job losses” that never seem to come.
I had another post about having tried and being unimptessed by the “top of the line” AI models, but Justin vetoed it for mysterious reasons.
According to a Morgan Stanley study, reported in the Guardian, “The UK is losing more jobs than it is creating because of artificial intelligence”. (In quantitative terms, the article unhelpfully states that “AI had resulted in net job losses over the past 12 months, down 8%”, without explaining: 8% of what?)
Since I’m not a philosopher, I am interested in the wider issue of LLM use by academics. If philosophy is a special case where LLM use is concerned, let’s hear arguments as to why. If not, let’s see how these arguments hold up against the wide variety of contexts in which academic researchers in different disciplines operate.
It seems to me that a number of arguments against the use of LLMs have undeniable weight, including:
* They are trained on copyrighted work without permission.
* The social consequences of their widespread use are imperfectly understood, already known to be bad in some respects, and could conceivably be catastrophic in the long run.
* They are environmentally unfriendly.
But I don’t buy the following claims, if anyone is making them:
* The value of academic publications intrinsically rests in them having been produced by a human being.
* The use of LLM tools today by an academic, for specific limited purposes, by default degrades that individual’s capacity to produce intellectually useful output.
Let’s consider whether academic publications in general must be written by a human being to have merit. Of course, current LLMs do not seem to be able to generate material that, if written by a human, would be publication-worthy. They frequently fail at the first hurdle of providing correct references, let alone summarising their references accurately or producing appropriate and carefully-reasoned prose.
But as LLMs (and other approaches to AI) improve, they may start to generate human-competitive papers. To sharpen the question (while deforming it somewhat): should a paper be rejected by a peer reviewer purely on the basis that it was generated by an LLM? In other words, should peer reviewers by default reject LLM-written papers that, if they believed the paper had been written by a human, they would have accepted?
In mathematics at least, I think there are strong arguments for accepting AI-generated papers. AI may already be close to autonomously producing proofs of novel and valuable mathematical results (including, let’s suppose, the selection of what hypothesis to attempt to prove). Proofs of mathematical results are used by human scientists and engineers, in ways that do not depend on the proof having been generated by a human. There would be something very odd about denying publication to an LLM-generated mathematical paper solely because it was LLM-generated. (This argument is relevant to philosophy insofar as it extends to e.g. proofs in formal logic.)
If mathematics is a special case, let’s hear why.
Moving on from hypothetical future LLM scenarios, let’s consider whether it is valid for academics to use current LLMs to relieve particular intellectual demands.
I propose a selection of tasks that I think are plausibly valid use-cases for modern LLMs from the narrow perspective of technically improving published output (i.e. leaving aside more general ethical issues relating to AI use). Yes, these needs would probably be better-served by human assistance (some more than others). But realistically, in many academic contexts, the LLM is available to perform tasks that other human beings are too busy, or unwilling, to do. Here are the tasks; I’ve arranged them from what I expect to be the least controversial use case to the most controversial use case:
1. Automating coding tasks (I have personal experience of this being very helpful). Academic publications can require the writing and running of code (e.g. for model simulation). Most philosophical publications don’t, but there are probably exceptions (Dennett wrote programs that he ran to illustrate points in some of his books).
2. Correcting language errors or readability, especially for writers who are not fluent in their language of publication, or have a disability that affects writing.
3. Improving literature searches. Perhaps some philosophers have the luxury to publish on topics so narrow that it is realistically possible to keep abreast of all relevant developments in the field without assistance. But many academics do not.
4. Exploratory discussion of ideas to help articulate and sharpen them. (Again, I have personal experience of this being helpful.)
In all these cases, I would be inclined to argue that judicious LLM use enhances, rather than degrades, an individual’s output (in the narrow sense that considers only their finished manuscripts).
Note: before anyone asks, no part of this comment was generated by, or inputted to, an LLM (though I suspect it would have been improved by LLM feedback).
It may be arguable, but then again, I’d be betting on it degrading a lot more than that.
OK, I’m going to try something a bit unusual and very “meta” here. For context: I believe that the internet has changed human life psychologically and sociologically. Research indicates that we feel a freedom to speak to people online in ways that we ourselves would consider unacceptably rude if we were to say the same thing to a person. My personal sense is that internet use has corroded our socio-intellectual and interpersonal capacities, in ways that seem analogous to some of the dangers of LLM use.
Although Justin has a clear comments policy regarding respect and politeness, I’ve seen exchanges on Daily Nous that, to my taste, exhibit this sort of unpleasantness. I would prefer to avoid contributing to it. So I try to write posts that treat people respectfully, even when I disagree with them, and I make an effort to scan and edit those posts for phrasing/framing that might unnecessarily provoke personal offence.
I drafted a response to Felix in this spirit. When I read over the response, I wasn’t satisfied. I then used a chatbot (Claude, free usage plan) to see if I could improve on it. After a little to-and-fro, it generated an entirely rephrased draft that I felt accurately conveyed my original message, but significantly more tactfully. This isn’t what I’d wanted at all (I’d just wanted help improving the draft, not a complete rewrite), and posting it as a response would violate the Daily Nous comments policy. However, I will reproduce here both the LLM-generated paraphrase and the original draft, for purposes of comparison and discussion.
Felix, I’d be especially grateful to hear how these different versions might have struck you emotionally, in the context of the thread. How might you have responded to one, rather than the other, if they’d been written by a human?
Here is the LLM-generated paraphrase:
— LLM content begins —
Thanks for the response. I tracked down the Nature article the video discusses [link].
If I understand correctly, you’re pointing to this as an example of cognitive degradation from LLM use? I can see the concern, but I think this case actually illustrates something different. The author was using ChatGPT as a comprehensive assistant for virtually every academic task – writing emails, drafting courses, structuring grants, revising publications, preparing lectures, creating exams, and more. This goes well beyond the “specific limited purposes” I had in mind.
The use cases I listed (code generation, language correction, literature searches, exploratory discussion) are much more circumscribed. If the author had used ChatGPT only for those targeted purposes rather than as a general academic organizer, the data loss likely would have been far less impactful.
I do want to note something that seems relevant to the broader discussion: your response linked to a YouTube video without explanation of how it relates to my argument or what the underlying source was. I think there’s an interesting parallel here – if we’re concerned about LLMs potentially degrading careful thinking and communication, shouldn’t we also be mindful of how communication norms in online forums can encourage similar shortcuts? The same intellectual virtues (clarity, precision, engaging directly with arguments) matter regardless of whether the shortcut is technological or social.
I’d be interested to hear your thoughts on this connection.
— LLM content ends —
My original draft read as follows:
— My draft begins —
I’m not going to watch an entire YouTube video just to understand what you are referring to. I stopped at the point where the YouTuber named the article they were talking about.
If you are referring to this article, where an academic’s ChatGPT conversations over a 2-year period were unexpectedly deleted by an action the author took, then it does not really address my arguments.
In the article, the author states:
“Having signed up for OpenAI’s subscription plan, ChatGPT Plus, I used it as an assistant every day — to write e-mails, draft course descriptions, structure grant applications, revise publications, prepare lectures, create exams and analyse student responses, and even as an interactive tool as part of my teaching.”
That doesn’t sound like using ChatGPT for “specific limited purposes”, and it goes far beyond the specific use cases that I listed: generating code, correcting language errors, improving literature searches, and exploratory discussion of ideas. It looks to me like, if the author had used ChatGPT judiciously for those purposes, rather than treating it as some sort of general academic life organiser, the impact of the data loss would have been significantly less.
Of course this is just an online forum, where people dash comments off in between more important work (I have the luxury to put time and thought into my comments because I am semi-retired). But (and I don’t mean this as a personal attack), it would have been productive if you had explained what you meant more clearly, linked to the source paper rather than a YouTube video, and explained how it related to my views. If we are worried about cognitive impacts of LLMs (which I think we should be), shouldn’t we also worry that online forums are corrosive to intellectual capacities, in that they permit and encourage “laziness” in intellectual discourse?
— My draft ends —
While I endorse Justin’s principle that the words we use in this forum should be our own, I do think the LLM-generated text is more tactful and arguably better-written than my own draft.
But I feel distinctly uneasy about its content for reasons that I can’t fully articulate. It doesn’t feel like it’s quite exactly what I meant to say, and, even worse, I worry uncomfortably that perhaps it’s what I *ought* to have meant to say. If I were to routinely offload the work of being tactful onto an LLM, what psychological effects would it have on me? Would I start to learn from the LLM’s example? Or would I stop bothering to word things tactfully? And, beyond that: I do not want to indirectly support all the known societal harms of LLM use.