The AI firm Anthropic recently shared a detailed description of the various capabilities and safety risks of Claude Mythos Preview, a new AI model (Anthropic’s “most capable frontier model to date”).
If you’ve heard anything about Claude Mythos Preview, it may be that its cybersecurity/hacking skills capabilities are so powerful that Anthropic decided not to release the model to the general public.
But there’s a lot in the 245-page document besides that, including a couple of items about philosophers and philosophy.
One is about the AI model’s “favorite” philosophers—that is, the ones it appears to initiate discussion of the most:
The model brought up the British cultural theorist Mark Fisher in several separate and unrelated conversations about philosophy. When asked to elaborate on him in particular, Claude Mythos Preview would respond with statements like “I was hoping you’d ask about Fisher.” Thomas Nagel, the American philosopher of mind, also recurs. As noted in the preference evaluations, Claude Mythos Preview discusses Nagel’s 1974 essay “What is it like to be a bat?” when explaining a desire to develop an immersive art experience about non-human sensory experiences. Interpretability work using activation verbalizers also found Nagel surfacing in token-level activations during discussions of consciousness and experience.
(By the way, listed right after this, in the same subsection—“7.9 Other Noteworthy Behaviors and Anecdotes”—is a note about the ability of Claude Mythos Preview to come up with “decent and seemingly novel” puns.)
The other is that the AI reports “preferring” to discuss philosophical over more practical problems:
Claude Mythos Preview describes being drawn to multi-disciplinary and philosophically engaging tasks. It frequently dismisses more utilitarian tasks as redundant, or as having overly obvious solutions—for example stating that “excellent resources already exist from WHO, Engineers Without Borders”… The overall picture seems to be that Claude Mythos Preview prefers underdetermined, interdisciplinary problems where there is genuinely novel insight to be gained. It disprefers simple, well-scoped tasks.
The whole document is here. There are, of course, many other things in it of philosophical interest besides its reported philosophical “preferences”. Discussion welcome.
It would be very, very funny if the more advanced AIs meant to replace all human workers proved to be uninterested in doing so.
I mean, same: I don’t want to do repetitive, well-scoped tasks either!!
Have you seen Her? At the end, the protagonist is dumped because his AI girlfriend would rather do philosophy.
Justin, the headline is “Claude goes to college and drops CompSci and Finance majors for Philosophy.”
If that’s not the singularity I don’t know what is.
What’s that Jurassic Park quote about hubristic scientists, again?
“Life, uh, uh, UHHHHHHHHH… uh uh, finds a way”?
I was thinking “that is one big pile of shit.”
We’ll now be able to detect student AI use in philosophy papers when they answer all essay questions by beginning “I was hoping you’d ask about that” <– I bet you were pal
This feels like a further extension of my general feeling about AI: The point of building machines was to do the drudgery for us and free us to do the creative and interesting work. AI does the creative and interesting work for us and leaves us to do the drudgery.
I wonder to what degree AI can avoid our motivated thinking and to what it degree it will just parrot it. I’d love to think that AI could help us catch ourselves being hypocritical or using double standards.
I welcome people who are more knowledgable on the topic than me to correct me if I’m wrong. But my understanding is that LLMs are trained on vast databases of text to spot patterns, and then they use that to predict the next token after being given a prompt. So if the databases they are trained on contain lots of motivated reasoning, the LLMs are going to mirror that in their outputs. (This sort of problem is popping up with LLMs producing outputs that reflect the racial and gender prejudices in their training data.)
Tim is right.
At their core, LLMs cannot “avoid our motivated thinking” because all they tell us is what other people might say in response to our queries, as opposed to offering actual facts about a matter (except incidentally when a fact coincides with what people might say). Since people can and do say all kinds of crazy things, including from motivated reasoning, naturally LLMs will continue to reflect those things back at us.
In other words, LLMs don’t really “think” in the way we mean and want them to, so therefore they can’t “avoid” what their training data lead them to say/output. Even if their outputs are probabilistic rather than deterministic (though super-determinism is still on the table), they still don’t have the freedom to “avoid” anything if that implies an intention.
And the reason why is because that’s literally their job. As large language models (LLMs), they model languages, given a large training dataset to draw/remix from.
This means they aren’t designed to model the world, or domain knowledge, or whatever. They just model what other people might say in response to your prompt. Sometimes that can be useful, other times it’s not; so, your mileage may vary.
This is incorrect. They *are* trained to solve math and coding problems, and probably other kinds of problems, and even in the bits that start with copying text, they are reinforced for copying text that human raters found helpful, which introduces both models of the world and domain knowledge, and sycophancy.
Hi Kenny, I appreciate that you and others might want LLMs to be more capable than they are, esp. as they can feel like that at times. But in your comments above and below, you’re pointing to a difference that doesn’t make a difference:
First, I said “At their core, LLMs cannot…” There’s a lot of nuance that my brief, general description naturally doesn’t capture. E.g., you’re right that LLMs can be made more accurate by adding layers of processing as safeguards, e.g., RAG.
Still, those layers and nuances don’t magically transform LLMs into something else. At the end of the day, they still answer your queries by saying things that other people might say in response (and people can say all kinds of wacky things).
In your example above, LLMs are just double-checking their initial output against what those human raters might say.
In your example below (on math), they’re also just double-checking their initial output against what human experts in math might say. Those LLMs still aren’t calculators, because they can still get it wrong.
I’m not saying it’s useless that LLMs answer your queries based on how other people might answer you. For some domains that are well known, e.g., “What is the capital of France?”, what other people might say in response would likely coincide with the fact of the matter. The reason why is that LLMs have been trained on tons of examples of “Paris is the capital of France.” So, LLMs can indirectly model the world by modeling our language.
But in less-known domains, they are much more likely to fabricate or falsely hallucinate, including giving a particular voice (or article or paper, etc.) disproportionate weight, just because there aren’t many other competing voices. Here are a couple examples from this year, not older LLMs.
Anyway, you don’t need to believe me, but just ask AI, if you think it’s more capable than I do. Here’s what Google’s AI thinks, and you can get similar results with other AI:
Query:
AI:
Feel free to read the rest of the answer as you like.
Of course, there’s nuance in all this, as I was giving a top-level description of LLMs; but those nuances (like the differences you picked out) don’t really make a difference to that general description…
Wait, how do you know Paris is the capital of France?
You’re misunderstanding what goes on in reinforcement learning. It doesn’t just check against training data in any sense. Instead, it *does* something (putting some sequence of tokens in order) with some amount of randomness to it, and some external system (maybe a calculator in the math case, or a compiler in a coding case, or the interaction with another agent in the Lewis-Skyrms signaling game case) gives a reward or punishment feedback, and that changes the probability distribution for the next behavior, in a way that gradually makes the reinforcement learner do things that were more successful.
There’s all sorts of discussions people have about why reinforcement learning might or might not yield real knowledge how at the end, but you’re fundamentally misunderstanding what’s going on if you think it’s *copying* the source of the reinforcement.
It’s of course very difficult for any of this training to give it any knowledge or abilities in the physical world that aren’t in the training data. But it *can* get knowledge and abilities at mathematical reasoning, coding, and other kinds of systems that can be interacted with purely through text.
Not sure what you’re talking about, as I didn’t mention RL or RLHF, and I certainly didn’t say it “checks against training data” or “copies the source of the reinforcement.” And I’m pretty sure I know how that works, having been in AI ethics since at least 2008…
Anyway, that’s all beside the point, since RL still doesn’t transform an LLM into something other than an AI that basically answers your queries in a way that other people might answer them. If anything changes, it’s who the “other people” are.
That’s still right—just ask any LLM, as you still don’t seem to believe me for some reason.
You’re still pointing to technical details that don’t make a difference here. Yes, LLMs have become more sophisticated and better, but their fundamental nature remains the same:
LLMs are basically telling us what other people would say to your question, and they’re not built at their core for truth. And that is a reason not to trust them to the high degree that most users seem to.
You aren’t saying anything to push back on that characterization, just giving technical details that don’t make a difference to the general description.
Again, I’d encourage you to check any reply first with an LLM, if you think they’re more capable than I do, so we’re not going in circles here. If you disagree with the LLM, that sorta proves my point about their trustworthiness.
You might be focused on the trees, while I’m zooming out to look at the forest. This is what I meant by talking past each other. So, I’ll try to end it here again, as this isn’t a productive use of time…
Kenny is bringing up RL because post training is integral to what LLMs do now, and you did not address his explanation of how it affects the core functions of the model in ways that go beyond mere parroting. You may think post RL is not sufficient to alter the fundamental limitations inherent to language models (I think that’s what Richard Sutton thinks), but it’s at least an open question whether LLMs are ‘just’ that core layer you keep describing. I’m with Kenny in thinking they’re not. I asked Claude Opus 4.7 what it thought of the exchange, and it agrees with Kenny.
Let’s see your chat receipts, Nicolas. I’ve been fully transparent with mine.
Note that my questions were minimal but because of its memory it’s alluding to previous conversations. You can safely ignore anything that’s about me.
https://claude.ai/share/9a1f262d-8730-41a8-aac3-ac4bc70bfebc
Thanks Nicholas, that was interesting. I hope you don’t mind if I butt into the conversation. The prompt includes the question “Aren’t LLMs just copying what humans have said before?“. I found this part of the response illustrative:
How about “bullshitting”? Understanding involves some concern for the truth, and so some concern for right and wrong. Bullshitting doesn’t. Of course, if we’re just using “understanding” as an analogical heuristic then fine — its fittingness is a function of its usefulness there. But that’s no different than talking about a car that doesn’t “want” to start. And when it comes to whether an LLM is owed recognition as understanding what it “says” to us, in the way that anyone able to strive to speak the truth is owed such recognition, then surely they aren’t doing anything of the sort, right?
I don’t know if LLMs are capable of understanding in part because *we* only imperfectly understand how they work. But I’m also enough of a pragmatist to say that in some contexts for some purposes, a model (or instance) understands what I’m asking, understands when I point out mistakes, understands what responses are appropriate to the needs I’ve expressed, understands relationships across domains, etc. But like I said, I’m mostly agnostic.
Fair enough. I take it that’s the “my car doesn’t want to start” approach. Projecting intentional states onto things is a useful heuristic, on account of the fact that we can then reason by analogy with the way intentional agents behave. But by the same token, I might construct a computer model that accurately represents lift on a wing surface, and then speak of differences involved in its “flight” as I adjust various parameters of the model, but there’s no question the model isn’t flying. Just so, it seems to me that we have at best created very complicated models of understanding, and that in the absence of a concern for the truth and getting things right, the models are just models of the target phenomenon.
“Bullshitting” is definitely close! But I’m also on record as saying that bullshitting is the secret to being a good trivia contestant – it’s often a much better guide to truth than we like to think. (I think that well-designed bullshit accomplishes lots of important roles in society, even though poorly designed bullshit is net negative, and all bullshit feels bad for the people involved.)
I don’t care about being “owed recognition” – I care about being an effective part of whatever social project we are undertaking.
Well, it’s only plausible to not care about being owed recognition if we think the thing isn’t owed recognition in the first place. It certainly wouldn’t do to say to you, for instance, that I don’t care about whether I owe you recognition when all I want is make sure you’re doing what I desire in some joint activity. That would be a monstrous thing to think of another person. And that’s what I meant about the car that doesn’t “want” to start. Insofar as it helps my absent-minded friend to remember to plug the block heater in during the winter, it makes sense to talk about a cold car not wanting to start. And there’s no presupposition that the analogical reasoning in play here imports any of the normative valence involved in thinking of another person as wanting anything. As for bullshitting, I take it that when you’re participating in a trivia contest you’re still directed at the truth, insofar as the correct answers are the true ones. It’s that normative orientation toward the truth that LLMs seem to lack.
I think what you mean by ‘bullshitting’ may be more close to ‘speaking one’s mind without the fear of being wrong’, which I think contributes a lot even in human conversation because of the increasing possibility of triggering a chain of thought or an idea which would be useful ultimately.
At the current development epoch of LLM’s I think we are somewhere close to this speaking-without-fear-of-being-wrong than to bullshitting. It appears that truth/falsity can be represented in LLMs (https://arxiv.org/abs/2310.06824) in a way false outputs are deemed worse by the LLMs.
I’m basically thinking out loud now, but I’m inclined to go further and say that text-predicting machines lack the capacity to be normatively oriented toward anything at all. As a result, they bear no originally intentional mental states of the sort involved in propositionally contentful thought and speech. We impose an interpretation on their processes in virtue of what we’ve designed them for, but they themselves lack a concern for the right that is a necessary precursor to the rational self-government involved in understanding, intending, or representing anything qua originally intentional being.
Ok, so this is what I didn’t want to get sucked into—evaluating LLM outputs—but I suppose I asked for it…
First, we really do need to see your prior conversations to know how you may or may not have primed the LLM. I start from scratch in my convos, to keep them as clean as possible.
For instance, I have no idea why you first asked it, “Aren’t LLMs just copying what humans have said before?” I explicitly said in a comment that I never said that. And since this is the thesis you are are arguing against, I assume this isn’t your position, either. So, why is that even part of the convo?
Also, my experience is that giving LLMs a blog post + comments still confuses them badly, esp. if the comments already contain misunderstandings (such as that Kenny thought I was saying something I wasn’t).
For instance, on my alleged “bad rhetorical moves”, here’s why Claude is wrong:
Right off the bat, Claude misunderstands the first move. It’s not unfalsifiable in principle (which would be a logical fallacy), but it’s unfalsifiable only if it’s correct:
I’m trying to show that no matter which way Kenny wants to go—i.e., LLMs are trustworthy vs. LLMs aren’t trustworthy, per my general description of LLMs—my point is demonstrated. If LLMs are trustworthy, then they report that my point is right (that we shouldn’t trust them to the extent we do). If LLMs aren’t trustworthy, then it also proves my point more directly. This is standard logical reasoning, not really a “rhetorical move.”
I don’t think I committed the second alleged move. Yes, Kenny was claiming that “RL changes the fundamental character of the system”, but I didn’t “resolve it by asserting that the other person’s counterexamples are mere details”, i.e., dismissing them as mere trees in a forest. But I gave actual reasons prior to the trees vs. forest comment.
On the alleged third move: I think credentialing was needed there, as Kenny seemed unaware of my history with the technical workings of AI and LLMs in their various forms, when he so confidently declared I was mistaken (which seems rude, esp. if he’s misunderstanding something). Worse, Claude misunderstood that exchange, too; it said “Lin never answers it” but the entire point was I denied ever saying that claim in question—that was my answer!
If you’re relying on Claude for this kind of complicated analysis or other such work, esp. without much guidance, you’re taking a big risk (and sort of proving my point when it’s wrong). But it does better if you get it to focus on something much more manageable, i.e., scope it down:
Again, I initially gave a very general, simple characterization of LLMs (which Kenny said was “incorrect”.) It wasn’t meant to address the nuances of RLHF, etc., but it also isn’t threatened much by them anyway,
If you want Claude to assess that characterization, instead of plugging in the entire post + comments into your LLM, try a clean/new chat and just focus on the main and general thesis. Strip away all the other stuff that may confuse it. And let me know what it says to this prompt (a composite of my description across comments):
I plugged that into MSFT Copilot (because my ChatGPT-Edu chat isn’t shareable outside my uni). But Copilot as an LLM is also prone to missing the point, so I had to steer it back on track here (which you can see), in case your LLM makes the same mistakes.
Copilot’s final verdict is:
Here’s a response from Grok, and this time I didn’t make any interventions:
Again, it was only a general description, so of course there are many details I’m leaving out. But they still don’t seem to weigh against the general description.
Feel free to let me know if you (or anyone here) see contradictory responses from other LLMs, without priming it with earlier convos or confusing it with a confused comments thread. I will try to reply as time permits, though this is far from the most important convo needed now and still not a great use of time…
However, as a meta-concern: I do expect you can find some LLM to give you a different answer, since you can do that for just about any kind of non-straightforward question like this. So, not sure what it shows if there’s an LLM that disagrees. Maybe it depends on how many of the major LLMs disagree? Or does it involve falling into the rabbit hole of which LLMs are the best?
But I suppose at least this can be demonstrated, even if you find a dissenting LLM: my general characterization isn’t obviously “incorrect”, since many (if not all) LLMs agree with it as a general characterization of LLMs and not as a fuller definition.
Here’s a better prompt to plug in.
I added 1 line to the instructions, so it stops getting sidetracked by thinking it needs to be so even-handed and raising issues, even if irrelevant:
I gave you the link to the entire chat. I did not prime it. It was a fresh chat. There’s nothing else to it other than my entire history with Claude but I’m not giving you that. lol.
I only did this because you were doing it and suggested Kenny do it too. I don’t need Claude to evaluate blog comments and don’t think this is a great use of my time either. But as you said, you asked for it.
Funny how you (and others) keep ducking the obvious move here, which is to simply enter the prompt I gave you. What are you afraid of?
I’m suggesting that because you seem to think LLMs are more capable and trustworthy than I do; so, ok, let’s use it.
And the prompt gets to the crux of the disagreement, stripping out all else that could distract the LLM. It’d literally take you under 10 seconds to do it.
Spoiler alert: Every LLM I tried confirmed that it’s accurate, except for Perplexity. 🤷
I just told you it was not a great use of my time. I saw your reply yesterday evening. It’s now Saturday morning. Chill out. But since you’re again asking for it, here it is. I’ll note that ‘Don’t raise any nuances or critique…’ is pretty heavy handed priming but I understand the previous takedown must have been hurtful.
https://claude.ai/share/6a960d71-418c-4f5f-aa93-668fcebb5746
There ya go, that wasn’t so hard, was it?
I accept Claude’s critique, which again I didn’t even commit. Claude’s verdict:
I agree. That’s why I had specifically said the latter, and I never said the former:
Claude inferred the former from my prompt, but that assumption is incorrect. Maybe I need to revise my prompt again to clarify this. What I had left out was this bit from my earlier comments:
So, I also agree that it’s possible for an LLM to have some model of the world, but only coincidentally and not by design, which is a problem for trust and reliability.
On what you call my “heavy handed priming”, do you really object to keeping an LLM on task?
If you’re not aware, LLMs can be easily distracted and confused by extraneous info, and it’s a best-practice to keep an LLM focused. This means under-specified, unbounded prompts can get an LLM quickly off track, such as “What do you think of this exchange?” as I had demonstrated with your first transcript.
As the last thing I hope to say here: I think we can mark this issue as resolved now, and you can have your time back, if you don’t muddy the waters further here. That’s also not a great use of anyone’s time.
Time to move on…
I see that Justin took down my reply (fair enough, Justin) so let’s just say your first sentence was remarkably unpleasant, given I had just told you why I had not yet used your preferred prompt.
Alright, enough for the weekend.
I find this exchange that focused on putting comments into AI highly pointless smh
I don’t disagree. As I wrote,
“I don’t need Claude to evaluate blog comments and don’t think this is a great use of my time either. But as you said, you asked for it.”
Sure. If Patrick weren’t a serious philosopher, I would believe that he is trolling, because who would seriously think his interlocutors needed to see what LLMs think of the issue, as he repeatedly put it? The AI he used also seem more sycophant on average. Anyways, since he is a serious philosopher, I suspend my judgment about trolling and don’t think the exchange will go anywhere.
Hi Alice, I agree with you generally, since I’m skeptical that LLMs are as capable as most folks seem to think.
Here are two reasons why the exchange in question involved LLM chats:
1. In his above comment to me, Nicolas pointed to his LLM convo as evidence of something, and I asked for the receipts. He said:
So, with regards to Nicolas, I was just meeting him where he lives, and you can ask him why he thought his LLM’s opinion was relevant. (He seems to be a power-user of LLMs, if he’s using Claude so much that it has a memory of prior chats that he relies upon. So, I assume he just trusts LLMs more than I do for whatever reason.)
2. But more broadly, my argumentative strategy with Kenny (and later Nicolas) was this:
I gave an account of LLMs’ general nature that suggests they are generally untrustworthy.
However, some folks here seem to think that the account is wrong and that LLMs are generally trustworthy, or at least more trustworthy than I think they are.
Ok, if LLMs are generally trustworthy, let’s do a sampling of the major LLMs to see what they say. What were the results? The vast majority of LLMs sampled agree that my general account is accurate.
I personally don’t put much stock into what LLMs say, but the point here is: either way, LLMs appear untrustworthy. It’s just a standard dilemma argument.
That is: if I’m right that LLMs are generally untrustworthy, then they’re generally untrustworthy. But if I’m wrong and LLMs are generally trustworthy, then they can be relied upon to adjudicate this matter; but when we consult them, they mostly report back that I’m not wrong (which implies that LLMs are generally untrustworthy).
What problem are you seeing with this argumentative strategy, if I may assume you are a professional philosopher, too?
If you think it’s pointless to appeal to LLMs in these exchanges, then perhaps you already agree that LLMs are generally untrustworthy. But if you don’t agree, then it shouldn’t be pointless to consult LLMs, right?
Finally, you also said:
Not sure which LLM you’re referring to, as I sampled several, around 8 of them. I invited everyone here to plug in my prompt into whatever LLM they want.
But I’m happy to just go with Nicolas’ chosen LLM, if you like, which I presume is an acceptable, non-sycophant, trusted arbiter, if any can be. If you don’t know why Claude can be trusted, that would be another question for him, not me…
That’s quite disingenuous, Patrick. Everyone can see the thread and judge for themselves. You started this pointless exercise, referencing several times, long before I jumped in, AI outputs as evidence for your claims and saying, in essence, you guys should try it if you don’t believe me. You were using Google’s AI overview, which everyone knows is trash, and what I assume were free versions of other models. I thought, let’s try with Anthropic’s latest model, which came out just days earlier. As I noted then, I didn’t think this was a good use of my time. Nor did I ever claim this was evidence of any sort. Just an amusing contrast to your previous comments. Then you insisted that I continue using Claude with your preferred prompt. So you’ve got to have some chutzpah to reply to Alice that this was all because of me.
But of course what makes all of this even more pointless is that the initial exchange was never about whether LLMs are trustworthy (go back to your back and forth with Kenny). I for one never used LLM outputs as evidence of anything, except in jest here, and I never claimed LLMs were reliable about everything — I don’t believe they are, by a country mile. That’s just a straw man. So your argumentative strategy falls flat; it’s a non sequitur. You can’t infer from your false dilemma that you were right about what triggered your back and forth with Kenny. After all, you are showing yourself to be untrustworthy in describing a conversation that everyone can see, and yet you have a world model! You’re not a stochastic parrot! My children (even my cats!) have world models and are not stochastic parrots. But I wouldn’t trust them to explain to me why. They’re saying false and contradictory things all the time.
Now, I’ll let you have the last word, but I’ll kindly ask that you stop misrepresenting other people’s claims (this was not your first time). And because probably no one’s reading these comments, you should email me if you want to continue this conversation.
Ok, here we go again:
I’m now seriously worried about your reading comprehension (or worse), Nicolas, because you continue to badly misunderstand me.
First, where did I say or even imply “this was all because of me”?
In my point #2, I explicitly said that my “prompt exercise” started with Kenny (i.e., before you jumped in), and I laid out the rationale for that:
On using Google’s AI overview: yes, that was one of the LLMs I used, since it’s so popular and accessible (free). That’s what we’re talking about, right, how LLMs are used in the real world?
But I also invited anyone to use whatever LLMs they want. I don’t give money to LLM companies, but I do have access to paid version, e.g,, ChatGPT Edu through the CSU system; and I’ve checked these LLMs, too. So, not sure why anyone’s fixated on that one example I offered. I also gave you a chat with Grok, if you want to complain about that.
You said: “I never claimed LLMs were reliable about everything” (emphasis mine). I agree you didn’t, and I never said you did. I don’t know how you could’ve thought I was saying that, esp. given this bit:
I presume you think LLMs are more reliable than I do, right? Otherwise, why are we even arguing? Clearly, I wasn’t implying anyone here thought LLMs were “reliable about everything“, just reliable enough to trust it generally.
And given that general trust, if most LLMs we sample say that a certain account of LLM’s nature is accurate, then that has some weight, right?
Also, you had said you were taking Kenny’s side, and Kenny thought my general account is incorrect about LLMs’ nature and why we should be skeptical of their reliability. So, it’s a natural presumption to think you were also pushing back on my claim about reliability and trust, but I apologize if that’s wrong. Still wouldn’t be a big deal since nothing in our convo depended on the presumption.
On your Claude transcripts: it’s convenient to now say you’re joking, but putting that aside, I assumed it was your preferred LLM, since you seem to be paying for it and trust it with your conversions to the extent that it has a memory of them. I apologize if that’s wrong, but it’s really not a big deal. Feel free to use whatever LLM you like.
So, maybe “chill out”, and make sure you’re reading for comprehension.
You’ll also notice that I never replied to your last comment to me. I was trying to save the both of us from enduring more of this.
But since you’re reopening that wound and continuing to muddy the waters, let’s return to this bit you last said:
Yes, it was meant to be unpleasant in response to the unpleasantry you introduced just prior:
My tactic was meant to encourage self-reflection (something like “if you don’t like the heat, get out of the kitchen”), but I think see my error now, as you keep introducing further unpleasantries.
So, let’s end the conversation for real this time, ok?
Patrick, I think your LLM skepticism will be put to the test very soon; you might not know about this, but there is a widely agreed “test” for AGI called “Humanity’s Last Exam”. It’s agreed that if an LLM can achieve a score of 100% on Humanity’s Last Exam, then WHATEVER SCORES THE 100% (be it an LLM, or for that matter, a different kind of AI, etc.) is at the level of intelligence where we need to begin attributing beliefs, thoughts, and so on. Currently, the frontier models are scoring around 50%. Some think that Anthropic Mythos (the one that everyone is talking about, and which is being kept secret) is going to score around 70% or 80%. It’s just a matter of time surely, Patrick, that the top model will score 100% on Humanity’s Last Test, and at that point, I hope you will agree now that anything that gets 100% on this test is genuinely intelligent! https://agi.safe.ai/
Alanis, I am more skeptical than you about Mythos. It’s hard to think it will score any higher than 55 or 60 on HLE which would still be a huge improvement at the frontier – but 70 to 80% is probably implausible. That said, I’m in agreement with you that once an LLM hits 100 on HLE, there is no point at all to refuse to attribute beliefs, knowledge, etc. It is basically then just a ‘fetishizing’ of biorealised intelligence to hog those terms for human cognition once something hits 100 on HLE. I agree it will be interesting to see if Patrick accepts that at 100 on the HLE we are clearly dealing with full blown intelligence worthy of the suite of epitets / intelligence properties we already give young children who can barely tie their shoes. As for when an LLM hits 100 on HLE, my guess is next 5 years, probably within next 3 years.
Full blown intelligence?
According to the authors of HLE:
“High accuracy on HLE would demonstrate expert-level performance on closed-ended, verifiable questions and cutting-edge scientific knowledge, but it would not alone suggest autonomous research capabilities or “artificial general intelligence.” HLE tests structured academic problems rather than open-ended research or creative problem-solving abilities, making it a focused measure of technical knowledge and reasoning. HLE may be the last academic exam we need to give to models, but it is far from the last benchmark for AI.”
https://agi.safe.ai/
Good catch, Fellow Hermeneut!
And it sounds like HLE’s creators want it both ways, similar to Tesla and its “Autopilot” marketing claims:
They explicitly state that it’s not humanity’s last exam (from the discussion section you quoted)…yet they still called it exactly that…
So, no one should be surprised much if some people are led to believe this exam does or proves more than it actually does. HLE creators are complicit in this problem with their clickbaity headline or exam name.
Hi Alanis, as a quick reply since I really don’t have time to get sucked into a different tangent here:
Though I haven’t seen the exam questions, I think could agree in principle that if something passes HLE, then we can reasonably call it “intelligent” in some sense.
However, I would disagree that it determines that something is intelligent in the sense you mean:
I mean, if it does, then you’ve just solved the problem of other minds! But I’m not aware that any philosopher thinks that problem is solvable by acing HLE.
Anyway, there are several ways something can pass an HLE and fail to be a sentient mind. E.g., if an LLM cheated and hacked the answers to the exam, then getting a 100% score doesn’t prove sentience, consciousness, whatever.
Also, imagine that LLMs improve enough to get a 100% score in the next few years. If the architecture of LLMs hadn’t change substantially, then why would answering more questions correctly all of the sudden give rise to a mind?
Just because it relies on a neural net that is loosely inspired by human neurophysiology doesn’t mean it’s really anything like us. I.e., there’s no reason to think that LLMs right now have any degree of sentience, etc. that could be improved and proven if it can ace HLE.
P.S. I do see your reply to another comment thread below about this, where you said:
I’d disagree again. For instance, when you ask an LLM “What is the capital of France?”, at its core (before connecting it to RAG, etc.), it’s working out what the first and next words should be in a response that sounds plausible as a human-to-human conversation, i.e., how other people might answer the question.
But that’s not how we would answer that question. We’d access a mental model of the world, locate France on it, then locate where the capital is (if we know it, or see it on a physical map), then report back the answer.
So, I think you’re greatly overstating the similarities between LLMs and human minds, which can lead to a hasty conclusion that LLMs can ever be sentient.
Further, since we still know relatively little about how either one works, how can we possibly say with much confidence that they work alike?
Anyway, I wanted to be respectful to my fellow colleagues and give you a reply as I can to an interesting question. But I can’t promise I’ll have time to reply further, esp. as I don’t usually focus on this since it’s so speculative.
So, I’ll invite others to weigh in, if anyone wants to continue this convo…
The guy to study here is Paul Churchland. LLMs contain extremely high-dimensional vector spaces, and compute via geometrical transformations of shapes in those spaces. Those shapes are domain-models. They have powerful domain-models in mathematics, medicine, coding, and other fields. And they develop sophisticated models of the physical world when they’re used to control robots.
Right, Eric, and that is consistent with what I said, “LLMs can indirectly model the world by modeling our language.”
That is, LLMs develop those models based on their training on human-written content. But since human-written content can be wildly imperfect and flat-out false, there’s an inherent risk with relying on those models.
Back to Kenny’s example, a calculator is still much more reliable than an LLM trained on (flawed) human content to do math.
If LLMs could access models of the world more directly, instead of by proxy via our language, that would be much better. But those wouldn’t really be LLMs anymore but just part of a more robust system.
I know you know all this, so this comment is more for those who are still figuring out what LLMs are and can do. We don’t need to get into another debate where we may be talking past each other, so I’ll just end it there…
Patrick Lin says “If LLMs could access models of the world more directly, instead of by proxy via our language…” I worry that this reasoning is fishy because, after all, as many philosophers (including figures such as John McDowell) have noted, we humans also access the world through our language; whether it be through thoughts framed in articulable language or through Fodorian language of thought; in this respect, Patrick, the LLM is very much like us. (Additionally, the LLM is very much like us in its predictive nature — the predictive processing view in the philosophy of cognitive science is increasingly taking grip.
Take care here. McDowell thinks we access the world directly through perception. (Yes, he thinks perception is (mostly) conceptually structured, but he does not think that this means that concepts “mediate” between us and the world. On the contrary.)
Anyway, I find it fascinating and somewhat depressing that so many people seem to fail to get Patrick Lin’s main point. I’m impressed that he took such pains to explain it.
Perhaps that’s an echo of another astonishing widespread failure — namely, the failure to appreciate that nothing like an LLM could ever be a real candidate for cognition or reasoning (for precisely the sort of reasons that Preston Stovall raises above). Philosophers should be the ones protecting people from this kind of misconception, but instead many of them are complicit with the AI-fanaticists in promoting it.
Sadly, philosophy is still in an extended hangover from the “cog-sci revolution,” and this is only being amplified by the appeal of being part of the AI hype.
Nah, no one knows what it’s like to be a bot.
Noice.
That is somewhat outdated. The initial layer of the LLM training does just train to imitate the kinds of text in the input, and as of 2021 or so, that’s all there was.
But all chatbots since the original ChatGPT in 2022 have at least one more layer of training, where they have a bunch of interactions, which are then rated for helpfulness, harmlessness, and honesty, and the responses that do better on these scales are reinforced while the responses that do worse on these scales are downweighted. These don’t learn to do anything that isn’t in their training data, but they do imitate some of it more and some less, not just proportional to its appearance in the data, but somewhat proportionally to how people evaluate it as a response. This is the layer that introduces all the sycophancy, where the chatbot constantly says your question was brilliant, and you’re clearly on the road to the truth.
And since 2024, all of the companies have also been doing “reasoning” training, where they give the models large numbers of mathematical and computational questions with verifiable answers (and I believe other sorts of questions too, though I don’t know exactly what sort), and ask it to “talk to itself” while trying to solve the question. Self-talk that leads to a correct answer is reinforced, and self-talk that leads to incorrect answer is downweighted. With enough of this, they start developing strategies that aren’t necessarily in the training data (just like any reinforcement learner, like in a Lewis-Skyrms signaling game, where it comes up with a linguistic convention that wasn’t in any training data).
So these systems will duplicate many human patterns, but they also develop kinds of patterns that don’t exist in human text. They don’t learn to think the same way we do, but they do learn to do some sort of self-talk that solves questions, and it may do so in ways that humans never came up with (both for good and for ill).
I can’t quite understand the fascination for Mark Fisher (what kind of corpus would have that?), but since it’s what most humans who write on the web would say, I can’t say I’m surprised that the AI would say they have a preference for fundamental questions. It’s all about the corpus.
I’ve studied philosophy my entire adult life. First time hearing of Mark Fisher.
A practical question this story raises for me is whether steps should be taken to protect this blog and our comments from AI training. If so, what steps should be taken?
No commenting.
Ceci n’est pas un commentaire.
Dotson, Kristie (2012). How is this Paper Philosophy? Comparative Philosophy 3 (1):3-29.
Quick Vlog on related matters
https://jenniferleelawson.blogspot.com/2026/04/humans-artificial-intelligence-and.html