A philosopher noticed something off about the paper he was refereeing for a journal.
In the following guest post, Jakob Ohlhorst (RWTH Aachen University) shares his experience and raises some questions about how to deal with the problem he encountered.
Reviewing an LLM-written paper
by Jakob Ohlhorst
It was bound to happen. I reviewed my first apparently LLM-generated paper.
I enjoy reviewing, helping the authors shape, sharpen and situate their papers. This wasn’t much fun though. I accepted the review by accident, clicking the wrong link; but the title and abstract were interesting enough, connecting different topics I am interested in. So I committed to it anyway.
Reviewing this text was an interesting experience. I started out curious about how the paper would fulfil the promises made in title and abstract (I’ll not go into any identifying detail, as I am concerned with the general issue, than the specific article). The introduction was fairly broad, the vagueness of some concepts and ideas concerned me somewhat, but I assumed that this was just giving the gist, playing fast and loose. The paper was short after all, I’d get to the substance soon enough.
The first section, introducing a special kind of coherentism was more or less right but underwhelming. Significant characteristics were missing. As a result the paper simply offered an explanation of coherentism. The second section was worse, however, as it was applying this coherentism to a specialised domain without saying exactly what was supposed to be coherent or how. Instead, the text pointed to general structural parallels between coherentism and the field.
By the third section, four pages in, I became aware of the prose. It was good English, but vague with a strange rhythm, exhausting to read. It felt like wading through cotton candy: little resistance, no real substance. Even the real-world phenomenon that coherentism was supposed to be applied to in section three remained underspecified and nebulous.
I pasted the most offending passage, the explanation of coherentism, into an online LLM-detector. The result was (65%) positive. My first charitable thought was, “maybe it’s just this summary which is laying the ground while the rest is more solid.” I checked the references. Those were hallucinated. I was quite angry.
For me, this raises a few questions:
- Is this one of the first cases of this in philosophy (and so the problem will only worsen); or has this already become widespread?
- Are editorial processes already flagging this pre-review, and I caught one that slipped through? Ought checking for hallucinated references be among the minimal tasks the editorial team completes before sending manuscripts to referees?
- How much can we rely on LLM-detectors?
What makes me optimistic: I was able to recognize the problem soon enough. But it was still a waste of time that I could have spent reviewing a real paper.
I sympathize with non-native authors who face considerable hurdles writing in academic English and who therefore consider relying on the help of LLMs. However, submitting completely LLM-generated text is unacceptable in terms of academic integrity, and not even a good idea prudentially, as the style is noticeable and not very agreeable. I would suggest using LLMs very sparingly, as even raising the suspicion of LLM-use in a referee will do a paper no good.
People sending in papers written by LLMs to such an extent should probably receive the same treatment as plagiarists. I wonder how this case and other cases will be handled.
Get a LLM to write the review and inform the author and publisher of what you have done.
Dead Internet Theory is depressing enough because, if it’s true (and it’s hard to ignore the possibility that it is), it means that a potentially incredible resource has lost much of its potential, and it will be hard to get back.
Peer review is one of the few quality control mechanisms in academia, and it already has a bunch of problems. There are already lots of things to review; the review process is vulnerable to normal human foibles; but it’s basically one of the few minimally decent and viable tools. Slop strains it to the breaking point. It’s already starting to spoil the sciences. Serious sanctions for authors who use it this way might be the only way to stop it. It has to be treated as *so* poor, so obviously fraudulent, that it’s effectively the equivalent of making up data in sciences.
Name and shame the fuckers. If someone submits a paper that can be shown, beyond a reasonable doubt, to have been written by an LLM, the journal’s editor should make that person’s name public, that name should be added to a list, and that list should be widely circulated and made publicly available in fora like here and Leiter. Zero tolerance, scorched earth.
(After being pretty strongly opposed to the last decade’s alleged uptick in public censoriousness, I’m not entirely sure what to make of my present bloodthirstiness!)
Ah, the good old anonymous urge to name and shame. Tar and feather for thee, invisibility for me.
God forbid we have different standards for blog comment sections as for peer-reviewed publications
I was asked to review a paper for a political theory journal recently. I looked at the paper quickly, decided that I wasn’t a good fit as a reviewer, and declined. Later that day I was using the copy I printed as scratch paper and I noticed an odd reference. It cited a chapter by a friend of mine in an Oxford Handbook. I was surprised that I didn’t know about this chapter, given that we talk semi-regularly about what we’re working on, so I looked at the table of contents of the book and no such chapter existed. There was a chapter by someone else on a similar topic, but the title and page numbers were different. The reference was clearly an AI hallucination. When I dug further, I found another case of the same thing. This was the first time that I’ve written a journal about a paper that I had already declined to review. When I told my friend about this, he said that he had previously reviewed the paper for another journal and pointed out the same thing, which means that even after the issue was pointed out the author didn’t make a correction. The journal did divulge to me that the author is someone without a university affiliation, but that makes the whole thing even odder because apparently the author is under no pressure to publish.
hey, let’s not go sowing suspicion of people without university affiliations
I certainly didn’t mean to suggest that independent scholars were more likely to engage in bad behavior. If anything, I would think that they’re less likely, insofar as some people who cut corners with publishing probably do so because they’re in danger of losing their positions and they’re desperate. I find the fact that this author used AI without being subject to that sort of pressure especially mystifying.
those who have lost affiliations temporarily (given the increasingly precariarity of academic jobs) also have the pressure of publishing so as to get back on the job market
Speaking as a journal editor (though I’d prefer to remain anonymous in this thread, as our AI policy is not yet prepared and public and I’m speaking purely personally), this is not unusual. We are receiving these regularly, and doing our best to find them before review (usually via hallucinated references).
But LLM prose is now good enough that it can be difficult to do so if the references are legitimate and the paper is in an area of specialty that doesn’t match that of the handling editor. For instance, the problem mentioned by the post author here is a common one – that the paper is saying superficially the right things, but in a vague way that really just summarizes the current literature. If one does not know that current literature well, that kind of vacuous paper can be hard to spot before review.
And further, LLM detectors are very often very radically wrong (detecting, e.g., text written long before the invention of LLMs as LLM-generated). I would strongly recommend against their use.
I am quite worried about what this means for the future of the editorial and reviewing pipeline, which readers here know all-too-well is becoming unmanageable even prior to widespread LLM use.
I agree that LLM-detectors should not be used as the criterion for exclusion. Rather it might be one of the detection heuristics for editors to have a closer look, just as the plagiarism detector does (mediocrely) in Editorial manager. Maybe watermarking technology will get better soon.
I also do not blame the editors at all; it took me four pages, and the paper’s prose was typical ChatGPT-speak. I fear that some papers will simply slip through.
I cringe every time I read/hear about a colleague using one to ‘catch’ and punish a student.
Well, my impression was that students do not even clean up their references. I’m sceptical about using them even heuristically in teaching because a false positive would bias you against the student.
Hence the hope that for editors it might serve as a heuristic. Referees should not need to resort to them either, I’d prefer. For me it was a checking whether my impression about the prose was right in the “heat of the moment” before I even thought of looking at the references.
“one of the detection heuristics”
It should be unacceptable to upload paper to AI detectors as to general AI, during the review process.
What are your concerns about the tools (beyond their unreliability and the ensuing risks mentioned already)?
The biggest obvious worry I see is that a lot of these tools take everything uploaded to them as training data for one or another neural net model. I suspect that things uploaded to an AI detector aren’t the best training data for lots of purposes, but that doesn’t mean that no company is trying to do this. When someone submits something to a journal, they expect that the journal will not send the text to anyone else except through the review, or eventual publication process. Sending it to third parties who might use it for other purposes, in order for the journal to check for AI content, would raise legitimate worries for some authors.
I see, I guess that could be avoided with running a non cloud/proprietary detector; they don’t seem to be huge algorithms.
Again, under the caveat that it just serve as a heuristic for further investigation, ideally not by referees.
Posting before the LLM apologists get in here arguing “no actually this is good”
I think I’m one of the standard “LLM apologists” here, but I don’t think there’s anything good about this (other than that the reviewer identified it pretty quickly). LLMs can be useful tools, but there are only so many things they can do usefully at this point.
I’m probably also one of the “LLM apologists” here.
But this isn’t an issue with LLMs, it’s an issue with a human who is committing fraud by claiming authorship for a paper which they didn’t write.
And that’s wrong, regardless of what anybody thinks about LLMs.
I don’t mean you in particular Eric, but some on DN have suggested that the problem is not the generation of papers by LLMs but the generation of bad papers. If true, maybe the author was actually just poor at prompting or their idea is not quite good (and they were reckless with citation) – the fact that their idea was put into writing by LLM is not the problem. Have better ideas!
FWIW I think cases like this should motivate a much stricter prohibition on using LLMs at almost any stage of the writing process. Certainly for writing out or improving ones phrasing of some philosophical idea.
I don’t think that having AIs generate papers is wrong or problematic. I generally object to bans, abstinence, purity, intolerance, etc.
The problems occur when paper authorship is presented dishonestly, or in violation of publishing norms. So, as many others have said, this issue can be settled by journals having AI policies.
If a journal allows, or requires, that papers be written by AIs, then it’s fine. (But then it would be dishonest for a human to pass off their work as AI work.). Conversely, if a journal prohibits AIs, then it’s wrong for humans to present AI work as human.
What we need are better policies for authorship.
Curious why you have pre-commitments against bans, etc. Why wouldn’t you judge each situation on its own merits to see if a ban could be justified?
We have bans all over the place—against murder, theft, assault, fraud, littering, unsafe driving, tons of other stuff. Do you not think those bans are a good idea, even if it doesn’t stop all instances of those acts?
And you seem to be proposing a ban yourself, i.e., against dishonest authorship. So, I’m skeptical that you’re actually against bans in general, despite your saying so…
Not Steinhart, but one can have a general, defeasible rule against bans. Why would that be a problem? Heck, some argue that most, if not all, moral norms are defeasible anyway.
I’m not concerned about defeasibility and am happy to agree that most (maybe all) moral norms and laws are defeasible. There can be excuses or justifications to do most things, including murder, theft, assault, fraud, littering, unsafe driving, tons of other stuff.
I’m asking about the principle behind a general objection on bans, etc., if there is one. (And I hope no one invokes libertarianism. If that’s all you got, then you got nothing.)
What libertarianism gets right is exactly that there is a defeasible presumption against bans. No one has the burden to demonstrate the need for exceptions to bans. One has to demonstrate that a practice is sufficiently harmful or risky to be banned. And because freedom is pretty important, the bar should be high. That’s the presumption. Suggesting that bans are generally okay unless you can prove they’re not is getting things backwards.
I didn’t say this:
I agree with this:
But this isn’t so clear:
Freedom is only one of many values. If other values are factored into the deliberation, then ok.
But if a standard-issue libertarian cares only about freedom uber alles, then that’s much too one-dimensional for something as complex as governing a pluralistic population with possibly conflicting interests.
Fair enough.
However, even if freedom is one of many values, it’s still up there. The bar should be high regardless. Bans can impose significant restrictions; they must be justified by sufficiently weighty considerations is all I’m saying. This doesn’t strike me as an implausible principle, whether or not it is a libertarian one.
When it comes to journals, I actually have no objection. Editors and boards are free to run them as they please. I was only commenting on your general question.
I think we’re agreeing on this, Nicolas:
But that seems very different from what Eric had said:
Maybe where you and I disagree is that I think that bar can be fairly easily cleared. The reasons are similar to those in my 14,000+ word critique on why AI shouldn’t be used by students, which I won’t rehash here:
https://emergingethics.substack.com/p/why-were-not-using-ai-in-this-course
Not to say anything for or against a defeasible presumption against bans, but what you are describing is not libertarianism. Libertarianism has no presumption against bans imposed by, say, journals. A journal is an organization run by people who (according to libertarians) have a right to ban whatever they want to ban so long as this infringes on no rights. Since nobody has a right to publish anything at all in a journal (nobody even has a right against anyone else that those others run a journal in the first place!) certainly nobody has a right that they be able to publish something in particular in a journal without it being banned.
Of course, you can be a libertarian and also think that it’s bad for journals to ban things, and I imagine many libertarians have that frame of mind, but many non-libertarians have that frame of mind too. We seriously confuse things if we act like libertarianism is all about being laid back and saying “anything goes” or whatever. What “goes” in libertarianism is not “anything,” it’s whatever is the result of people exercising their rights. If people exercise their rights to ban all sorts of stuff in the context of (e.g.) journal publication, then that is what the libertarian must accept. They can whine about it and say this is a bad idea, but their whining is no more libertarian than the whining of a non-libertarian who also opposes bans, of which there are plenty.
Someone who really hates bans could propose, say, a government law forbidding journals from banning certain things. That would be a ban, but a ban that bans bans might be better, ban-wise, than no ban. Whatever the merits of such a view, obviously it’s not a libertarian one!
I wasn’t describing libertarianism. Nor was I commenting on the journal issue per se. I was only responding to Patrick’s swipe at libertarianism. I only said it got something right. I’m happy to frame it in terms of rights. The presumption is still that I may not interfere with your rights—for instance by banning a class of acts—without a good justification, for instance that such acts violate some other rights. Freedom of association is a perfectly good justification. And I certainly didn’t claim that no one had a right to ban. That’s absurd.
Yeah, those are excellent questions.
I tend to think of bans as imposed on acts that aren’t clearly morally wrong.
Like bans against alcohol or cannabis, or against racial intermarriage, or against gay sex, or banning certain books. So, murder is forbidden, but I’m not sure I’d say it’s banned.
Bans (like the ones I mentioned) have never worked out, and they generate more collateral damage. Specifically, they generate all kinds of cheating (e.g. black markets, or clandestine groups, corruption).
And bans lead to cultures of purity and intolerance.
So I think bans produce societies with are both more punitive and more corrupt. And there’s good evidence that this happens even with AI bans in classrooms. The prof becomes a cop, the students become more devoted to deviance.
And I think none of that is good.
What’s the difference between forbidden and banned? It’s hard for me to imagine a plausible argument that things forbidden by law aren’t bans…
I agree that some bans have backfired spectacularly, e.g., war on drugs. But it’d be hasty to generalize that to other bans.
A ban on AI isn’t so much a moral issue, as is the case for many of your examples. Yes, it can be unethical to use AI esp. when it’s prohibited by an instructor, but the case against AI use by students (for instance) is more about practical effects and risks.
Similarly, the case against AI-written submissions to academic journals is also a practical one, not so much a theoretical or moral one.
I need to get up at 3:45am tomorrow to catch a flight, so I’ll bow out here. Have a good week as you can, all. 🎃
Some examples of banned/forbidden things that aren’t clearly morally wrong: driving on the left side of the road, driving without a seatbelt, submitting taxes after April 15th, professors having sex with their students, etc. etc. But it seems plausible that many or all of those things should be banned/forbidden. And there are a *lot* of things covered in that “etc. etc.”.
The excellent AI papers are not getting caught because they are, well, excellent in all the ways that matter, including citation accuracy, elegant and precise prose, and so on. We can still ask if journals should accept them. For instance, fraudulently passing them as yours, if you’ve barely done any of the work (unlikely if the paper is truly excellent in all the ways that matter), is in clear violation of most journal policies. The question is where you draw the line of authorship, an important and difficult question to be weighed against the epistemic goods we might be passing on by weeding out those papers.
I don’t see why a case like this would motivate the idea of a prohibition on the use of LLMs at various stages of the process. The current case seems to be one that motivates the idea that it’s a *bad idea* to use an LLM carelessly to try to improve one’s writing, because it might make it worse. But if it really is possible to use an LLM to make a paper better, I think it’s great if people do that (just that most people aren’t good at using most LLMs in that way).
If the logic of “guns don’t kill people, people kill people” is flawed, then so is the logic of “LLMs don’t plagiarize, people plagiarize”.
Actually it’s not.
I hope everyone sees this phenomenon for what it is: the beginning of the end of academic publishing and a grave threat to graduate education as such. It’s one thing to have to retool undergraduate classes to avoid AI. But this represents a devastating blow to a 5000 year-old project, where human beings try to write down and share their best ideas with each other in a public forum. Once the channels over which we share ideas become sufficiently polluted, this ancient project will effectively end. It doesn’t matter how much integrity you display as an individual scholar; the readership won’t be able to trust that the average source is human, and that’s the end of the whole thing.
Trump could only dream of inflicting this much damage on academia.
Soon, the LLMs will no longer hallucinate sources. The machines already have paid/pro versions which source arguments perfectly well. Soon, there will be no easily detectable prose style. If you review regularly, by Fall 2026 you will be reviewing LLM-generated papers and job market writing samples. It’s hard to even imagine how inhuman academia will be in 2030, on this trajectory.
As a recently tenured person, I honestly feel like the very last person who managed to scramble into a lifeboat.
Some people think that what is worthwhile in academic publishing is not the fact that a human composed the words on the page, but rather the thoughts the words on the page provoke in readers. I think this is reflected in the term “publishing” and its etymological connection to the word “public”. (Of course, “publishing” in this sense is probably at most as old as the printing press, because before then it was hard to share written work widely. Certainly academic journals are younger than that, and not obviously the be-all-and-end-all of what publishing could or should be.)
To me, the question of whether reviewing LLM-generated papers and job market samples is good or bad depends on whether people have learned to use LLMs to generate good or bad papers. If you’re imagining a future where the papers actually produce good, useful, and interesting thoughts in readers, I think that’s great – and I suspect that the humans directing the LLMs will still be very essentially involved for many years. But if you’re imagining a future where the papers are actually sloppy and bad, but take hours of reading to recognize that fact, then that is awful – and I definitely see that as a possibility.
If you want to have your mind stimulated, and you are comfortable having an LLM play a role in that, why can’t you just talk to the LLM? Why on Earth would journals need to be involved in this? Even if you abstract from the role that journals play in the livelihoods of young academics, it seems a complete waste of resources and time to have LLM material sent through the peer review process when you can just talk to the LLM yourself and get the mental stimulation.
Journals are a solution to a communication problem. For many of us it is the communication that is important. For you the communication seems like it was always secondary. You just wanted access to ideas that were not your own. Once upon a time if you wanted ideas that were not your own, you had to rely on other people, and that meant relying on journals. If for some reason I do not understand you are happy for the ideas that are not yours to also not be anyone else’s, what LLMs represent is a way to get what you want without journals being involved. To continue to rely on journals seems like relying on a solution to a problem you no longer face.
The reason why I don’t just talk to the LLM is that I haven’t figured out how to get the LLM to generate the good ideas! If you can get an LLM to generate good ideas on topics that are of interest to me, I want to hear them.
Just because an LLM generates a good idea once doesn’t mean it’s going to generate that idea again. If there is a good idea somewhere, we should communicate it to people. I agree that communication is the point! But I don’t care whether the idea came to someone in a dream, or was produced on an acid trip, or came about through years of slogging in a library, or was generated on a ouija board, or was pure human genius (whatever that means). What I care is that the good idea doesn’t just remain confined to the one time and place where it first originated, but instead gets shared.
The point of the journal system is to identify good ideas and communicate them, not to gatekeep about how people are supposed to generate them.
“The point of the journal system is to identify good ideas and communicate them, not to gatekeep about how people are supposed to generate them.”
I think it is a way for people to communicate their ideas. What you are describing is not another way for that to happen. It is people sharing something a program they bought made. Your relationship to an idea generated by your LLM is closer to the relationship the institution which gives a researcher a grant has to the researcher’s ideas, than it is to the researcher’s relationship to their ideas. Perhaps I am wrong, but I do not think the Templeton foundation submits articles for publication written by their grant recipients.
You also present a false dichotomy. Not publishing LLM generated ideas in journals does not mean they stay private. You can just publish them online, on your own blog or tweet them. Anyone interested, as you are, in such things can easily find them there. Indeed, the barriers to access are lower with such a route to publicity.
That’s interesting – this response seems to deny a lot of the distinction between discovery and justification. I don’t think anyone has ever particularly cared whether the ideas a person published came as a result of conscious thought, or bubbled up during meditation, or came from a wild dream. Why would they care whether it came from consulting a computer? It seems strangely parochial to me to think that certain ideas shouldn’t be published in journals, not because they’re uninteresting, or harmful, or unsupported, or written poorly, but rather because of where they came from.
If you thought that LLMs are generating *bad* ideas (especially bad ideas that look good long enough to get past a reviewer) then I could see a very good reason to be unhappy at the thought of LLM publishing. But the person I was replying to earlier in the thread seemed to be thinking that it would destroy academia if LLMs became *good* at pursuing truth and knowledge and providing it to people.
I think that one challenge here would also be that we expect a degree of either consistency or progressive evolution of ideas across a philosopher’s corpus, and there is value in studying that longitudinal development. There is no guarantee that AIs generating texts would have that meaningful consistency and continuity through the development of an idea and its explication across fora. Certainly, it would seem to be folly to ask “what was the AI thinking/intending when it said that?” as we might ask of an Aristotle or Kant.
I, for one, am simply not interested. If computers get better at philosophy than human beings and journals start filling up with such papers, I simply don’t see the point.
I think a key point is that we need to rethink statements like “computers are better than humans,” especially when it comes to philosophy. LLMs are not competing with humans in that sense.
For one thing, LLMs and their outputs are products of human minds, like essays, artworks, or films. Any philosophical accomplishment an LLM achieves ultimately traces back to the people who built and guided it. I understand competing intuitions about this reason though. Maybe
as a test case, think about Watson, the chess-playing computer. When it wins a match, should we credit Watson itself, or the engineers and programmers behind it?
but the better reason is pointed to above, LLMs still rely heavily on human input, prompting, and maintenance to produce meaningful work. So any output they generate is really a collaborative effort between humans and machines, not a competition for human thinking.
I see your point though- a world where professional philosophers act like my students and just blindly paste prompts and copy outputs without engaging in their own philosophical thinking inbetween sounds horrific and boring.
I’m not sure I agree.
Suppose that in 30 years, almost every film is made by artificial agents. An artificial agent investigates what human beings like and generates prompts that are given to another set of artificial agents that generate storylines and eventually whole movies. Then the first agent assess how they did, and improves the process.
I can imagine that these films would be absolutely amazing, just getting closer and closer to perfection every year. Maybe you can trace them somehow, through the training data, to human creativity.
But the thought leaves me cold. To me, I’m describing a dystopia.
How do you feel about a world in which AIs are producing scientific and mathematical papers much faster and better than humans originally did, and this is leading to huge technological advances?
I accept that scientific advance is a good thing (even though I’m sceptical about the idea of mathematical progress by way of AI).
In this scenario, is the AI producing the papers, perhaps with some prompting, or is it a fully interactive process where the scientists are in control of (almost) every step?
If it is the former, I find the thought pretty bleak, even if it lead to technological advances. It evokes in me feelings of meaninglessness and nihilism (here, I’m merely reporting on my own feelings, not making an argument).
If it is the latter, then I suppose I could be convinced that it would be a good thing, but it may be that we are already losing that level of control.
But what does ‘better’ and ‘perfection’ mean in all these scenarios? Is everyone in tune with some ‘intuitive meaning’ of these ‘progressive’ terms that I somehow failed to be privy to?
I’m not sure why I can’t get by with the intuitive understanding of them.
We speak of progress in science and mathematics all the time, and while those things are hard to pin down exactly, and people try to work out what we do mean or should mean by them, I don’t think the precise definition, whatever it is, makes a difference as to whether the use of AI in those fields will be for the good or not.
Or am I wrong?
Not sure if this was in jest, but it is precisely this ‘intuitive understanding’ — somehow divorced from the decades of problematisation in philosophy of science and history — that, well, I don’t know what it is. And yes, an understanding of ‘progress’ is central to claims about X contributing to progress. What exactly is “the intuitive understanding” of progress and whose intuitions are being appealed to here?
Yes, but it is the proponent of AI use in science, mathematics and philosophy that relies on a naive understanding of what progress is.
I don’t think I need to spell out what progress is, because I am not interested in *that* kind of progress, whatever it is.
Either it is progress, and then I’m not interested, or it is not progress, and I am not interested.
Ultimately, however, I would probably go for the latter disjunct, arguing that whatever progress the proponent of AI thinks they are getting isn’t actually progress at all.
But my point is I don’t have to.
I reject the premise that AI will soon be writing groundbreaking papers.
If we should proportion our beliefs to our evidence, we have no reason to believe it will. See: the premise of this post.
Yeah, I don’t endorse that premise either. I think at most, we have good reason to believe that in the next few years, many people will be using AI tools in ways that help them write somewhat better papers (though there will also be an extremely large number of people using AI tools in ways that don’t help them write better papers, and some using them in ways that make their papers worse).
But I think it’s the goodness of the paper for readers that matters.
The 5000 year-old project has not relied on people reading journal articles written by unknown people. For most of that 5000 years, it relied on knowing (and to some extent trusting) the person or people who wrote what you’re reading, or at least trusting the editors of the publication you’re reading to know (and to some extent trust) the person or people who wrote what you’re reading. No reason to think that’s endangered, right? (Which is not to say that we won’t lose something valuable by losing the current journal system.)
It’s interesting that we don’t really know who composed some of the greatest outputs of that 5000 year-old project—the Epic of Gilgamesh, the Iliad and the Odyssey, or the Tao Te Ching. Or, controversially, the scriptures. Authorship is of little importance to us when we assess the value of these works. Maybe human provenance is—maybe—but why?
Right, but in each of those cases it was hugely important to us that people we knew and trusted were vouching for the outputs.
In many cases vouching meant fabricating a legendary identity for an unknown or nonexistent author, so trust there was, but trustworthiness I don’t know. I think the content was significant enough—played a culturally central role—that an author *had* to be made up (or a sacred source, in the case of the scriptures). I agree with you that we’re not losing any of this even if we’re losing the journals, in part because authorship in the modern sense is a very recent phenomenon in human history and was not considered nearly as important as it is to us now for much of that 5000-year project.
Of course there is! AI is already disrupting all of the knowledge and trust that you describe. Since the vast majority of output is now computer-based, the knowledge and trust that other humans are producing the material is dropping precipitously. The more people who casually admit to AI use, the more that people will lose their sense that they know who is creating what. That’s the point.
I suspected as much, but it’s good to see it from this perspective.
Why are people submitting papers written by LLMs? Surely, the work of LLMs is not (yet) good enough to be published.
Lots of people submit human-written papers that are not (yet) good enough to be published. The fact that LLM-written papers are much easier to produce makes it more tempting to submit them, even when they are far below standard. (Especially for people who haven’t been in contact with the philosophical community, and thus don’t have a sense of what the standards are.)
The same reason they submitted them to Clarkesworld, and thereby almost destroyed the magazine. The opportunity cost is low, the reward of success fairly high.
I find myself wondering the same sort of thing about people who make and post AI videos of animals doing implausible things (or other stuff) on social media. Mostly they look not very good, so there’s not even skill in prompting such things displayed. But even if they looked good, what does the person get out of it? Does it still make them feel good to get “likes” even when they didn’t really do anything? It’s very alien to me.
The fact that most of them don’t look very good should indicate that the few people who do manage to relatively frequently generate good ones really are doing something. Just because an AI was involved doesn’t mean there was zero skill or work done by the human, any more than the fact that a printing press was involved doesn’t mean that the author of a book did nothing.
Maybe? But I guess I still don’t see what the value is. It’s interesting to see, say, a dog actually do something silly or amazing, but it’s not interesting to see a simulacrum of a dog do something amazing, any more than it’s amazing to see a cartoon person do a free solo of El Capitan, even if it took some skill to make a good cartoon.
Maybe the reply here is to compare these videos to special effects in movies, but even there, if there’s not a story, the effects seems dumb to me, and in these videos there’s not really a story. If people can use them to effectively replace cartoons, that would be interesting in a way, but I don’t think that’s how people see them. I simply don’t understand the appeal or understand what people think they are doing. To me it looks much more like a person lip-syncing to a song and pretending they are singing.
Yeah, this seems like an appropriate response. Just seeing an interesting visual image can be really neat if you’ve never seen something like that before. But once you’ve seen a few of them, it’s a bit lacking unless the visual image can be in service of something more.
Also, I’ll add that a person lip-syncing to a song can be an important part of a great performance! Even though I’m not a fan of drag shows usually, I have seen some that are great, usually because the lip-syncing is in service of a broader performance. (And also, in these cases, everyone is usually aware that the performer is lip-syncing, rather than thinking the performance is great in part because it seems like the person is actually singing.)
Yes, sometimes lip-syncing is the performance, though of course it’s a different (and I’d guess I’d say obviously lesser) type of performance than singing. But again, that doesn’t seem to me to be what is going on in at least the vast majority of AI videos posted, or, in a different way, AI papers submitted. It seems to show a sort of emptiness, a desire to get credit when one hasn’t done anything, or at least not what one has purported to do. That’s why even if Milli Vanilli did the best lip-synicing ever done, they still would get no credit. And most AI videos seem worse than that, because at least with Milli Vanilli, _someone_ was actually singing, but in these videos, nothing that purports to have happened actually happened.
Obviously there are downsides to it, but this gives a strong reason to move back to authors’ identities being known to the editors as a stop gap (i.e. against triple blind review). The efficiency cost of having LLM generated materials in the reviewing pipeline could be quite devastating to an already strained system.
Genuine question: how would this help? Is the idea that people would be less likely to submit junk if their name is on it?
I had a similar experience a few months ago as an associate editor. I got a paper forwarded from the Editor-in-Chief, and from the title and abstract it sounded interesting. From the first few paragraphs, I thought it sounded like a scientist who wasn’t totally familiar with philosophical literature or philosophical style but had some interesting connections they wanted to make, and I was looking forward to seeing the main argument, and whether it was going to require some revisions but enable a cross-disciplinary connection. However, as I got farther into the paper, where the arguments should have been, it started dissolving into bulleted lists, and it never got much beyond the basic suggestion of the idea that was mentioned in the introduction. I don’t recall whether the citations were all familiar ones that were clearly correct, or if I just hadn’t checked the citations – the only real confirmation I had that it was LLM-written was the fact that the body had so many bulleted lists, and maybe some features of the prose style. It had taken me about 30-45 minutes to read this, write a brief desk rejection note, and then write a longer note to the Editor-in-Chief about what had happened.
This particular case wasn’t too bad, but I do worry about a situation where we start getting LLM-written papers that are at the quality of mediocre borderline human-written papers, and the journal system gets even more flooded than it already is.
The “optimistic” response by reviewers and editors would be to get much quicker with summary rejections, but this isn’t that optimistic for authors, who will need to get better at writing papers that clearly get over this bar. The pessimistic response would be for the system to get even more jammed up than it already is.
I do think there are ways that many people can use LLMs to help them write better papers, and some people already are. (In my case it’s mainly by helping me write code to draw diagrams and do simulations that help clarify my ideas for readers.) But the easy way to use them doesn’t make the papers good – just gets bad papers (or interesting paper ideas that the author hasn’t actually thought about) much closer to mediocre.
I can’t believe it had bulleted lists. That suggests the person didn’t even *read* what they’d submitted.
If there were some way to actually *prove* a person had used AI to write a paper (unfortunately, there isn’t really), I would want such a person blacklisted from all journals.
I reviewed an obviously LLM-generated paper a few months ago. It contained several hallucinated references.
In the future, when I do peer review, I will always look at the references first.
While it’s a way to save yourself the frustration, this kind of sleuthing is not an efficient use of your time as a reviewer, I’d say. After all we’re not trying to check for plagiarism either as a reviewer.
That’s why I was happy to see that editors implementing/developing protocols to catch them before anyone loses time looking at them (see the comment by executive editor below).
If the paper violates the journal’s AI policy, then I’d think it’s appropriate to ban the author for a few years, if not permanently. AI slop is an existential risk for media and other publishers.
If the journal doesn’t have an AI policy or is fine with AI-written papers and wasting the time of peer reviewers, then that’s on the journal, e.g., if they see a decline in willing reviewers given this reputation.
It’s not hard to have an AI policy, which isn’t set in stone and can evolve. But an absence of a policy just opens the door for all kinds of trouble…
This is the way.
The threat of such a ban will hopefully dissuade those in the profession from using LLMs and then trying to pass off the work as their own. For such a ban to work, clearly editors need to be justified in believing the paper is AI-generated, and the author needs to be informed of the decision and have the ability to appeal it.
It will also help reviewers feel more inclined to accept invitations. I spend a lot of time providing people with constructive feedback when I write reviews, especially when I see the paper needs a lot of work and suspect it is from a junior member of the profession. I don’t want to waste my time reviewing AI-generated papers. I think that within the year declining to review for journals without an LLM use policy, or perhaps even those who don’t enforce some sort of sanction or ban on offending authors, will be a reasonable move.
It would be nice to have a system that prevents the paper from being reviewed at other journals, so other people don’t also waste their time on the same paper as well. Not sure whether that sort of thing is feasible. Maybe a private message board just for the editors of journals to share information such as the paper title, author, and reasons for flagging it as LLM-generated?
I do think it’s hard to have an AI policy that can be used in the way you want, but also isn’t going to cause a lot of problems. Policies in general are hard. I suspect it’s better to leave this up to human editorial judgment, maybe with a very tiny policy like “authors judged by the editorial team to be abusing the submission process (e.g. by submitting obviously bad work in a way that jams the refereeing system) will be banned from future submission”.
Why do you think it’s hard, and what problems are you anticipating? Would they be better or worse problems than we’re currently seeing, i.e., a stream (or flood) of AI-written submissions because there’s no AI policy with consequences?
I’m not necessarily saying that the policy should be a full AI ban. though that’s what I’d do if I were running a journal. E.g., a policy could allow for AI use, as long as AI is acknowledged; if AI’s contribution is so substantial that it’d merit co-authorship if it were a human-made contribution, then I’m on record arguing that the AI should be listed as a co-author for the sake of transparency, contra most journals’ policies: https://ethics.calpoly.edu/AIauthors.htm
Your proposed “very tiny policy” also seems fine. I might even like that better since it’s much broader than just an AI policy (and thus no so tiny?).
Any reasonable policy that directly or indirectly addresses AI use and communicates consequences would likely be better than no policy.
Yeah, I’m thinking that policy is “tiny” because the policy itself does very little work – it’s all the humans who exercise judgment that are doing the work. Some people think of a “policy” as something specifying clearly what is or isn’t allowed. I’d rather judge actions by the consequences than by whether they follow some pre-specified rules (since we inevitably won’t write the rules perfectly).
Cyber security professional (trained in philosophy and logic, still reads sites like this to “keep up”) here. This is exactly what people are proposing in the bug reporting space, particularly for open source projects without sponsors. The maintainers of the “curl” application (a commandline web client, etc.) are so sick of slop reporting that they are going “nuclear”. The problem is that at last news, *even this isn’t working*. I worry that the journals aren’t far behind. As for motivation (mentioned elsewhere), I cannot help but wonder that in both the academic case and the software case whether or not there may be (for example) state actors sowing chaos on purpose, in addition to the never ending parade of people who want the scraps.
My anecdatum: I’ve completed over twenty referee requests this year (23, IIRC), and haven’t encountered this yet (nor in previous years, either).
I’m sure it’s coming, though.
I was talking to two journal editors in the summer. They both noted that the number of their submissions recently more than doubled – due to LLM-generated papers. But the kicker is that they are now getting reviews that were LLM-generated. Machines are writing it, and machines are reviewing their own writing. Who needs humans!
If it violates a journal’s AI policy, reviewers who use AI also should be banned as both potential authors and reviewers for at least a few years, if not permanently.
Something dramatic needs to be done to stop the bleeding as much as possible.
Using AI to review is arguably worse, since it likely requires uploading the paper to an LLM, which now allows the paper to be used as source material for it and perhaps other LLMs – against the authors’ consent and without attribution.
“wading through cotton candy” is so beautifully put.
My view, for which I will not argue here, is that if someone submits an LLM-generated paper to a journal, not only should they be permanently banned from writing or reviewing for that journal, but the journal editors should notify the “author’s” department and perhaps even the university authorities who deal with academic misconduct. The fist should come down hard and everyone should know it will come down hard. Otherwise, we will all be wasting a lot of time playing cop, just as many of us already do with lazy students.
Couldn’t disagree more. LLM-generated papers that aren’t fit for publication should be rejected. Once they are fit for publication, they should be accepted. We should be thinking as a discipline about how to make LLM-generated papers fit for publication, not trying to ban the possibility.
I do think there’s a potentially big problem brewing if people start producing large numbers of LLM-generated papers that aren’t fit for publication, but that take an hour of some editor or reviewer’s time to identify as such. It’s fine when people submit a 35-page screed in all-caps that makes logical fallacies every three lines – you can screen those out easily. The problem is that LLM-generated texts that don’t have highly skilled human guidance can easily get to the level of a mediocre paper that takes a long time to deal with.
Quick fix: charge for submissions (and pay the editor/reviewer)!
Charging for submissions helps some things – but I worry that it doesn’t do much to resolve this particular problem. Human authors have always had to pay in time and attention spent preparing a manuscript. When the problem is that some people submit twice as many papers by spending half the effort writing and editing them as others, a moderate financial cost might sufficiently deter them from flooding journals.
But when the problem is that some people suddenly have the ability to submit 20 times as many papers by spending only 5% of the time prompting an LLM to write a mediocre paper, then that opens the doors to people who took a few philosophy classes in college, then went into tech and made a few million, to now come back and start flooding the journals in a way that a small fee isn’t going to dissuade.
To clarify: my suggestion is a large fee, sufficient to adequately compensate the time cost being imposed on the assessors. (As discussed in the comments there, this would probably work best if departments were willing to subsidize their grad students for at least a submission or two each year. Or concerned faculty might contribute to a philanthropy to support members of the philosophical community who would find this economically burdensome. Regardless, fixing the referee crisis seems more important to me than ensuring that there are no economic barriers to submitting papers for peer review. Having free access to a completely broken system wouldn’t be worth much.)
Or concerned faculty might contribute to a philanthropy to support members of the philosophical community who would find this economically burdensome.
Somehow having some members of the community be objects of charity for others doesn’t really seem like a happy “solution” to me. I think we should look elsewhere.
Better suggestions are always welcome. I’d just caution against letting the perfect be the enemy of the less bad. (It may be wishful thinking to imagine that any entirely happy solution is possible, in which case your dismissal risks leaving us with a disastrous status quo because any improvement remains imperfect. As a general rule, I recommend tentatively supporting the best-seeming policy proposal currently on the table, until a better proposal is offered to surpass it.)
As a general rule, I recommend tentatively supporting the best-seeming policy proposal currently on the table,
I guess the difference between us here is that the proosal you make seems like a quite bad one to me, worse than going on like we are, and so not worth trying at all.
If someone wants to start a journal of LLM-generated papers, let them do so, or if a journal wants to give some space to papers that are LLM-generated, let them do so. My proposed policy (if that’s what it is) applies to papers submitted as if they were written by the person submitting them but which were, in fact LLM-generated.
Permit me a human reaction, which I will not defend and on which I will not elaborate: fuck all this technological bullshit.
And yet you wrote that on a computer connected to the internet.
MFW the quantifiers are restricted.
(Alt reply: And yet you live in a society, curious…)
Next time blame it on your counterpart who lives in a non-actual possible world where they don’t have computers.
Never! (Present disagreement aside, I loved More Precisely.)
Hey, I don’t mean to be snarky – but we’re all caught up in this technological mess. I share your frustration (I grew up among the Amish, I got a strange relation with tech). But we gotta figure this thing out and it ain’t gonna be easy.
There’s a Turing test problem that we ought to be focusing on here: How soon will it be before AIs can write papers that will pass the most stringent peer-review and that are utterly indistinguishable from human-authored papers? One year? Two? Five?
I think we’ll pass this Turing threshold soon, and, after we do, the old model of journal publishing is over. Journals can still be committed to publishing the best work, but now all (or almost all) the best work will be done by AIs. (And, inversely, AIs will probably be better article reviewers than humans. No need for humans in the loop at all.)
After we cross the Turing threshold, using journals as gauges for hiring, retention, tenure, and promotion will be useless. Top-journal prestige will disappear. But then one of the main professional features of journals vanishes. And there won’t be any need for journals as media of communication, because we already have better media for that.
The journal model hasn’t been around forever (it’s mostly a 20th century thing, with a few exceptions). We used to do philosophy mainly by writing letters. Maybe in the future we’ll do philosophy by training specialized AIs. Or in other ways.
I suspect it’ll be quite a while before unaided AIs can produce great papers, but I wouldn’t be surprised if, some time in the next year, some human is able to cleverly use an AI to do most of the work in producing a great paper. I suspect the time frame between the first primarily-AI-generated great paper and the time when AIs can easily and routinely generate great papers will be more like the time between the first self-driving car that safely completed a several mile race and the time when self-driving cars easily and routinely do it than like the time between the first AI that regularly generated fluent grammatical sentences on arbitrary topics and the time when AIs easily and routinely generate B- level undergraduate writing.
What were you thinking the time “between the first self-driving car that safely completed a several mile race and the time when self-driving cars easily and routinely do it” is? LLMs and Google are telling me different things. Sometimes I’m seeing 15 years. Sometimes 20. Sometimes I’m seeing that we aren’t there yet.
Yeah, that’s the range of uncertainty I’m talking about.
TL;DR. AI is advancing in ways that are extremely fast and truly astonishing.
All my timelines about AI are messed up. I worked in the early AI industry around MIT in the 1980s when I was a computer scientist. I went through the first AI winter, but I was studying neural networks pretty seriously by 1985, and I used a neural net in my dissertation (for which I wrote a program that generates and interprets metaphors). I then went on to do AI work in NYC, and got a patent. So I think I have some experience in this field. But so what. Pre-pandemic, I remember thinking that Kurzweil’s 2045 date for human-level AI was far-fetched. Maybe 2100. Then transformer architectures emerged, and AI took off in ways I would never have thought possible. And far, far faster. Now I think maybe anything’s possible. Settle the Riemann hypothesis next week. Solve the continuum problem next month. I have PhD level conversations with GPT-5 and I feel like I’m the stupid one. I’m sure I’ve gone too far in my assessment of future progress. But mainly I think we could have ASI long before we’re prepared to deal with it.
Yeah, I think “my timelines about AI are messed up” is one thing that we should all agree to!
I don’t mean to be claiming anything too strong with “I suspect the time frame … will be more like … than like …” – just literally that this is what I suspect.
And I know what you mean about Kurzweil – for a long time, 2045 seemed impossibly soon, but right now it feels impossibly far away (given that it’s as far in the future as the moment when I first saw phones with QWERTY keyboards and first started watching 480p streaming video is in the past).
To add to the anecdata: I recently flagged a paper that I suspected was AI-generated. The paper was superficial, encyclopaedic in its recitation of the literature, and its thesis was unsupported by the arguments, which were so thin as to be virtually nonexistent. A spot check of a citation found it to also be nonexistent, despite the author (“author”?) providing quotes from the supposed source.
At this point, I stopped checking for further evidence of LLM generation, since the paper’s quality was such that it would be rejected, anyway.
I suspect that this was a case of an academic for whom English is not their first language, based on information I was privy to as editor of a special issue. I’ve known others in similar situations admit to using LLMs to help clean up language or even “polish” translations, but this was my first experience of a journal article that I suspected to be AI-authored in some significant degree. I hope it’s my last, but I’m not optimistic.
It would be an interesting experiment for some AI enthusiast to create a journal (online, presumably) specifically for the publication of AI-generated papers. It could require authors to be completely transparent about their use of AI, including perhaps making the LLM a co-author. It might not publish very much at all now, since not many AI-generated papers are likely to be of publishable quality, but that might change over time. It might not eliminate the incentive for authors to try to get AI-generated papers in traditional journals under false pretenses, but if anything it would make it easier justify negative judgments about them for doing so since they would have an alternative outlet available where deception wasn’t necessary.
How to think about this depends on what your expectations are for how AI develops.
If (as I am on balance inclined to think) AI might become more useful for writers but will not ultimately be able to produce work at the same level as a strong philosophy paper (any time soon) then this is not really a serious problem. Academic submissions are not anonymous at the editorial level, and it is fairly easy to think of policies that will prevent individuals from overwhelming a journal without doing serious harm to the scholarly enterprise (e.g., no-fault six month break from submitting after a rejection; longer ban if you commit gross scholarship errors like hallucinating data; if necessary, restrictions on submissions without a verified academic email address.)
If instead AI actually gets to the stage where LLMs can produce work indistinguishable from the best human-written philosophy (but faster), then we’re back to the issue that these discussion threads keep coming back to: is the point of philosophy research the fruits that it creates, or the process itself? People like me and (I think) Kenny Easwaran think it’s the research output that matters; we react to ‘what if LLMs produce philosophy indistinguishable from human-generated philosophy’ rather as we would react to ‘what if LLMs produce cures for cancer indistinguishable from human-generated cures for cancer’. But of course this isn’t the only attitude you might take; the LLM debate just brings to the surface longstanding questions about what philosophy research is really for.
I’ll put myself on the same page as Kenny Easwaran here: it’s the output that matters.
But there’s another variable: what kind of output?
Lots of philosophical research involves finding patterns in the archive, or asking the archive a question, or incrementally extending patterns in the archive. And I suspect that those aspects of philosophical research have already been or will soon be automated.
So maybe humans will do some different kind of philosophy — human philosophy will become far more like art, wilder and more creative. And this might make philosophy more accessible to the public.
Obviously, I’m making guesses here. But as AIs get stronger, I suspect a division of labor will emerge between machines and humans.
Your view strikes me as reasonable, but I nonetheless think there is a value-relevant disanalogy between the outputs of scientific inquiry and the outputs of philosophical inquiry, and that this difference is important when it comes to AI-produced philosophical work.
Many of the outputs of scientific inquiry arguably contribute to human welfare in a more direct way than the outputs of philosophical inquiry. Suppose an AI were to produce a cure for cancer, but the biological mechanisms involved are so complex that no human could possibly understand them. We surely would judge that this is an important scientific achievement, even though this would not contribute to the expansion of human understanding of the universe.
As I see it, the main value that philosophy offers is that it contributes to human understanding in some way (though exactly how it does this is hard to characterize, given widely recognized issues about philosophical progress). This may be a bit hard to imagine, but suppose that some AI were to produce a work of philosophy which is too complex for a human to understand (to make this more plausible, suppose that it is on a topic like phenomenal consciousness which many humans find to be an intractable problem). Also suppose that if we were to understand it, we would recognize that it is a valuable philosophical achievement. If the AI lacks the capacity for understanding, and instead only mimics understanding-like behavior, it seems to me that this philosophical work would be of no value whatsoever.
This is an extreme example, since it requires an unlikely scenario in which an AI produces philosophy which both is (seemingly) valuable and is not understandable by humans (though this view probably applies mutatis mutandis to work in theoretical physics, where it perhaps is more plausible that near-future AI will produce seemingly valuable work which cannot be understood by humans). A less extreme example is one in which AI produces work which is not read by anyone. At present, there are plenty of philosophy papers which get published which are barely read by anyone, but at least they make some difference to the mental state of the author (I take it), and thereby offer some small contribution to human understanding. If journals were glutted by AI-produced work, I imagine this would lower incentive for humans to read the contents of those journals. In such a (fairly plausible) near-future scenario, my feeling is that any unread AI-produced philosophical work published in those journals would have no value whatsoever. If the glutting of journals by AI-produced work would lead to overall less reading of philosophy by humans, the consequence would be disvaluable when it comes to human understanding of the world.
You had me up to the end. I’ll grant that if we held the situation relatively fixed and AI led to a glut of bad philosophy destroying the journal system, the situation would be worse for philosophical understanding. But I’m not convinced things will remain fixed in that antecedent event. What we might see instead is a return to the practice of philosophy as dialogical interaction, group conversation at live events, and public-facing work of the sort that matters to people outside the discipline. If this were to happen, Nous and Phil Review could remain important metrics of philosophical insight and skill, so the journal system need not collapse entirely, but they would be much more widely treated as defeasible, and perhaps given less weight in the balance.
I have a sense that something like this is already going on, as the committees and events I’ve attended over the last two or three years appear to be shifting toward more focused and protracted emphasis on philosophy as conversation, and less on the rote exposition of a 50-minute idea or 20-page article. I may be wrong about whether this is happening, but if it is that strikes me as something good for the discipline, and for human understanding.
Sure, my more substantive view is earlier in the comment. I agree that if philosophy shifts toward more conversation-focused activity, that could be mostly fine when it comes to human understanding. Though I would maintain that a lot of serious philosophical work occurs primarily in written form, such that the decline (even if it is only partial, as you suggest) of this area of philosophical activity would be a loss of some value to humanity. Speaking personally, I feel like my thinking is fundamentally different (and better) when writing than when speaking, and that my ideas benefit from the rigorous review process of publishing in academic journals (even though this process is often frustrating).
I will also mention that for personal and professional reasons I would prefer that philosophical merit is judged primarily on written research output, and not primarily by in-person exchanges. This is because the latter probably tends to favor people who are better socially connected, and personally I am working in China, making travel to international conferences rather difficult.
One more thing: you mention a shift toward more public-facing philosophy. I’m curious if you think there is some reason this sort of philosophy (when written) would be less prone to AI replacement than more academic philosophy. I would think the opposite, since public-facing philosophy often consists of summarizing views and arguments which have already been established in a more academic context. Maybe you are thinking of youtube videos and such? I also suspect that AI will soon be able to produce media of this sort which is at least as good as what is produced by humans.
Agreed about the value of writing our thoughts down, and the value of journals as metrics of philosophical merit. As for public-facing work, I’m skeptical of AI’s ability to replace things like essays in the Atlantic, or the NYRB. But setting that aside, I was thinking more about events and lectures where philosophers interact with an interested public — something like the salons of 18th-century France, or the townhall lectures of people like Emerson in the U.S. in the 19th century, or Dewey in the 20th. Part of this is just supply-and-demand; as online fora (and journals) get flooded with AI slop, the increased demand for the alternative will incentivize and support a shift in the supply side. Anyway, it’s little more than a hope or dim intuition, but my sense especially during and after the COVID lockdown is that things are moving in this direction.
I suspect that there is a good chance that there are researchers already testing this out systematically, Sokal hoax-style. i.e. generate and submit a large number of LLM papers and see how they do across a range of academic journals. The broad purpose being to test whether our assumptions that they will do worse than human authors is justified.
We first need to be clear about what “AI-generated” means. It’s possible for a paper to be AI-assisted, where the ideas and arguments come from the author but AI is used to refine the writing or structure. That’s different from having AI generate the arguments or analysis itself. At this point, AI can’t produce genuinely new ideas, so a paper written that way will almost always be poor.
Given that, the most practical way to assess whether a paper might be AI-generated is to check whether any references or quotes are hallucinated. Using AI detection tools is largely pointless. They produce too many false positives and negatives, and even acceptable tools like Grammarly use AI and can be flagged. More importantly, uploading or pasting a manuscript into a detection tool raises the same intellectual property concerns as feeding it into any other AI system. Reviewers have an ethical obligation to protect the confidentiality of the work they review, so those tools shouldn’t be used at all.
Of course, checking for hallucinated references is only a temporary fix, since AI will eventually stop making those mistakes. And because most models still can’t handle generating full-length journal papers, people are likely creating them section by section. The real challenge for editors and reviewers is to distinguish legitimate language assistance from actual idea-generation and, just as importantly, to recognize the difference between AI-written text and a simply bad paper written by a human. I suspect once hallucinated references don’t happen it won’t be practically possible.
I think checking for hallucinated references is a very bad idea for two reasons.
One, which you allude to, is that LLMs don’t have to make these mistakes. My experience is that if you turn the ‘research’ setting on, you get rid of 90-something percent of mistaken references, often well over 90%. And the ones that are left are often mistakes that humans make *all the time*, especially repeating a mistake from the literature about what the details of a paper are. (It often happens that a paper gets mis-cited by someone prominent early on, and this mistake gets widely repeated. The most common occurrence of this is with identifying the last page of a paper in an edited volume. When everyone makes this kind of mistake, the LLM often will as well. So it’s not like it’s perfect. The same is true on those rare, but not unheard of, cases where PhilPapers is wrong – both humans and LLMs will just take it as settling the matter.)
Now the free version of ChatGPT only lets you do 5 research queries a month, so anyone trying to create something with the free ChatGPT might get caught unless they use up their 5/month. But there should be any number of other ways to detect someone who is using the free version of ChatGPT; it’s a long way from the quality you’d expect from a professional.
The second reason this is a bad test is that humans make so many mistakes in bibliographies. If I rejected every submission with a bibliographic mistake I’d have almost nothing to publish. The test will have false negatives *all the time* unless you allow for a fair bit of slack in what counts as a mistake rather than a hallucination.
Putting those together, we’re already at the point where anyone who is paying for LLM usage can do better by this test than a large number of humans writing papers in the traditional way. It’s just not a good test.
Getting some of the page numbers wrong in a real reference, or citing it from the wrong journal, or from a year before or after it was actually published, and occasionally even reporting the title of a paper as the famous line in it, are the sorts of mistakes humans make. I take it that when people mention “hallucinated references”, they’re talking about the phenomenon of citing a paper that really doesn’t exist at all.
But of course, LLMs are getting better at not doing that.
> The test will have false negatives *all the time* unless you allow for a fair bit of slack in what counts as a mistake rather than a hallucination.
OK, so allow for a bit of slack. The kinds of hallucinations we are talking about aren’t subtle errors of the kind where you get page numbers wrong in the way that many others have. We are talking about egregious hallucinations where papers or authors that simply don’t exist are cited as if they do. I don’t buy that distinguishing between clear cut cases of this and cases that we should let slide is really that challenging at the present moment.
I take your point about human citation errors, but what I’m referring to isn’t that kind of mistake. Anyone who’s spent time working directly with ChatGPT knows it still fabricates sources outright or merges several real ones into a single plausible reference. That’s not just a typo or a repeated citation error. It’s a structural issue in how the model predicts text. The “research” setting helps, but it doesn’t actually verify sources; it only improves the odds that a citation happens to be real.
So while checking for hallucinated references isn’t foolproof, it’s one of the few practical signals available at the moment. Once those disappear, I agree that reviewers will have to fall back on assessing whether a paper shows genuine understanding or just surface-level synthesis.
I have to say, even though most people seem basically in agreement, I find the lack of a more or less complete consensus on this issue depressing. Obviously this is my own view, but it seems to me that someone who has no qualms with LLMs producing what appears in philosophy journals do not have any understanding of the point of philosophy. The purpose of philosophy is thinking. Thinking requires a great deal of labor. The LLMs will never be able to think, since their processes are fundamentally rooted in repeating what has already been said or done before, and thinking involves calling the models, assumptions, methods, concepts, etc. that came before into question (i.e. thinking can never be automatic or rote). This means at one and the same time that human work is infinitely more valuable in philosophy than the work of LLMs since it is the only source of real thinking, yet that it will likely be very very difficult for human work to distinguish itself by its outward merits since genuine thinking is hard and rare and the project of a long period of time or gestation. What this means is that LLMs can easily drown out and destroy the incentives for producing what is valuable in philosophy by providing a much easier cheap imitation of what is (to borrow a turn of phrase from Nietzsche) a very slow-growing fruit. Every philosopher in my opinion should be deeply opposed to this development because they should be opposed to laziness, thoughtlessness, regurgitation, and the destruction of institutions that recognize and reward deep human effort and labor. I don’t really see an interesting debate here, only an obviously destructive force that some are bizarrely welcoming and cheering.
No one here has “no qualms with LLMs producing what appears in philosophy journals.” Everyone seems to agree that passing off LLM junk as your own work is bad on at least two counts: it’s fraudulent and it’s junk.
As for the point of philosophy, I’d say it would be concerning if suddenly philosophers agreed on what it is. We don’t—I for one don’t agree with you. This is the least surprising—and least depressing—fact about philosophers.
I did not say some people are unconditionally accepting of any use of LLMs. I assume most if not all people want no fraudulence and no junk, as you say. But they may still have no qualms, given those conditions are met, with “LLMs producing what appears in philosophy journals,” exactly like I said. Qualms and conditions do not have the same meaning. I also did not say I think it would be good for every philosopher to agree with the point of philosophy is. I said it is depressing when people have a conception of philosophy that is open to things (LLMs sourced material being published in journals) that are so at odds with what I consider the point of philosophy to be (genuine thinking). I assume that similarly if you found a not insignificant number of people who thought the point of philosophy was simply to appear smart, you might also find that depressing, and it would not follow that you think “all philosophers should suddenly agree on what it is,” only that it is depressing that people are so far away from caring about what you think actually makes philosophy valuable. It is kind of exhausting that so many responses on this site are like this one and only a series of uncharitable misreadings.
Uncharitable misreading of what? You alluded to a lack of consensus that I’m replying should not be surprising. You implied that some of us have no qualms and I replied that we do. No one thinks publishing AI content doesn’t raise any important ethical, scientific and philosophical questions.
You say “The purpose of philosophy is thinking.” Sure, we all like to think. But then what’s the point of publishing? Why have journals (or books or substacks or other texts)? You thunk, now you’re done. Socrates probably would have agreed with you, and he didn’t write anything down. I can sit in my armchair and think all day, no need for journals and their pesky reviewers.
And if we’re looking for arguments for P, and an LLM comes up with a good one, one that’s valid and plausible, or even valid and sound, surely the fact that it originated with an LLM doesn’t impugn its validity, plausibility, or soundness. A good argument is a good argument no matter who or what comes up with it.
Yes Socrates is a perfect example. Do you believe the thinking Socrates did was pointless? Did Socrates think by himself, or did he practice dialectic, dialogue? Are journals in which human beings respond to one another not an example of dialogue? Whoever said thinking is a solitary activity in which you engage with the thinking of no one else? What an absurdly impoverished understanding of thought. I look for thinking in philosophy, not mere validity, soundness, or plausibility, that is, I look for something “thought provoking” in the sense that it opens up new avenues of thought. While arguments can be produced through automatic rote processes, the provocation to thought that in my opinion is at the core of all the best philosophy cannot be
Neither thinking (in your sense) nor dialog requires journals. And you can have a thought-provoking dialogue with an AI.
I don’t think that the point of philosophy is thinking. I came to philosophy to get closer to truth, not to think. Thinking is just something I have to do if I want to use philosophy to hunt truth.
While this is an important topic to discuss, I think we need to discuss what norms and expectations are being fostered in graduate programs. The pressure to publish is strong; next-gen philosophers have (probably) greater familiarity with AI than many of the people commenting here, including myself; and my impression is that high schools and universities send mixed messages about AI use, with some trying to incorporate AI and many trying to teach “responsible AI use”.
If philosophy graduate programs do not have clear, enforceable AI rules/policies, then next-gen philosophers will continue to have unclear messaging about the place of AI in research writing. Sure, there is an expectation on display here in the comments that one should not submit an entirely AI-generated piece of crap–everyone agrees; but what about using AI to double check grammar or clarity or to help flesh out an idea one is struggling to put into words?
In my sub-discipline, some journals welcome manuscripts whose authors have used AI to some extend so long as the authors are transparent about it; other journals outright refuse to consider a manuscript whose author has used AI in any manner; some journals have no clear policy. The messaging is all over the place!
I think a uniform AI policy adopted by all graduate programs is sensible and needed. I imagine a world in which the APA, in conjunction with other philosophical associations, publishes a clear AI policy that is then unanimously adopted by all graduate programs. (If one school bans AI use altogether while another promotes responsible AI use, will students in the latter program have a publishing advantage? If a school lacks an AI policy, and some of its students use AI while others do not, do the former have a publishing advantage? I assume the answer to both questions is probably yes, assuming that AI is used productively and not to generate slop.)
It’s not clear to me that we need a uniform policy on the use of AI in philosophy any more than we need a uniform policy on appeals to empirical science or a uniform policy on the use of acronyms for important premises in arguments or a uniform policy on how we evaluate papers vs books as measures of professional merit or a uniform policy on whether a priori metaphysics is worth doing. There are many different things that philosophers want to do, and thus many ideas as to whether these different things are good ways to do it.
We shouldn’t want one fad to take over the whole field, so we should have a variety of venues with different policies on these things, so that there is space for diversity, even if one method makes it really easy to publish a bunch of papers that make progress in one paradigm.