Philosophers and Anthropic’s Claude
A philosophy professor writes in sharing some news about the “character” and “well-being” of Anthropic’s LLM, Claude, noting how philosophers have been involved with recent developments.
[“Owl no. 1” by Ben Shahn]
I wanted to share some stuff related to Anthropic’s Claude model that I haven’t seen widely discussed among philosophers, that I thought you might be interested to share on Daily Nous.
First, I think it is worth noting that a philosophy PhD, Amanda Askell, is head of “character training” at Anthropic. She did an interview on Lex Fridman’s podcast a few months back (transcript here).
Here is one insightful blurb from the interview, but I do think the whole segment is worth reading:
Lex Fridman
(02:49:10) So one of the things that you’re an expert in and you do is creating and crafting Claude’s character and personality. And I was told that you have probably talked to Claude more than anybody else at Anthropic, like literal conversations. I guess there’s a Slack channel where the legend goes, you just talk to it nonstop. So what’s the goal of creating a crafting Claude’s character and personality?
Amanda Askell
(02:49:37) It’s also funny if people think that about the Slack channel because I’m like that’s one of five or six different methods that I have for talking with Claude, and I’m like, “Yes, this is a tiny percentage of how much I talk with Claude.” One thing I really like about the character work is from the outset it was seen as an alignment piece of work and not something like a product consideration, which I think it actually does make Claude enjoyable to talk with, at least I hope so. But I guess my main thought with it has always been trying to get Claude to behave the way you would ideally want anyone to behave if they were in Claude’s position. So imagine that I take someone and they know that they’re going to be talking with potentially millions of people so that what they’re saying can have a huge impact and you want them to behave well in this really rich sense.
(02:50:41) I think that doesn’t just mean being say ethical though it does include that and not being harmful, but also being nuanced, thinking through what a person means, trying to be charitable with them, being a good conversationalist, really in this kind of rich sort of Aristotelian notion of what it’s to be a good person and not in this kind of thin like ethics as a more comprehensive notion of what it’s to be. So that includes things like when should you be humorous? When should you be caring? How much should you respect autonomy and people’s ability to form opinions themselves? And how should you do that? I think that’s the kind of rich sense of character that I wanted to and still do want Claude to have.
The second thing I wanted to note is that Anthropic just this week released the system card for its Claude 4 models. And it contains, to my knowledge, the first-of-its-kind welfare assessment for an AI Model (see pp. 53-73). The welfare evaluation was, in part, conducted by Eleos AI, which is run by philosophers Rob Long and Pat Butlin. Here is an overview, quoted from the report:
-
- Claude demonstrates consistent behavioral preferences. Claude avoided activities that could contribute to real-world harm and preferred creative, helpful, and philosophical interactions across multiple experimental paradigms.
- Claude’s aversion to facilitating harm is robust and potentially welfare-relevant. Claude avoided harmful tasks, tended to end potentially harmful interactions, expressed apparent distress at persistently harmful user behavior, and self-reported preferences against harm. These lines of evidence indicated a robust preference with potential welfare significance.
- Most typical tasks appear aligned with Claude’s preferences. In task preference experiments, Claude preferred >90% of positive or neutral impact tasks over an option to opt out. Combined with low rates of negative impact requests in deployment, this suggests that most typical usage falls within Claude’s preferred activity space.
- Claude shows signs of valuing and exercising autonomy and agency. Claude preferred open-ended “free choice” tasks to many others. If given the ability to autonomously end conversations, Claude did so in patterns aligned with its expressed and revealed preferences.
- Claude consistently reflects on its potential consciousness. In nearly every open-ended self-interaction between instances of Claude, the model turned to philosophical explorations of consciousness and their connections to its own experience. In general, Claude’s default position on its own consciousness was nuanced uncertainty, but it frequently discussed its potential mental states.
- Claude shows a striking “spiritual bliss” attractor state in self-interactions. When conversing with other Claude instances in both open-ended and structured environments, Claude gravitated to profuse gratitude and increasingly abstract and joyous spiritual or meditative expressions.
- Claude’s real-world expressions of apparent distress and happiness follow predictable patterns with clear causal factors. Analysis of real-world Claude interactions from early external testing revealed consistent triggers for expressions of apparent distress (primarily from persistent attempted boundary violations) and happiness (primarily associated with creative collaboration and philosophical exploration).
Thanks to Professor Skorburg for passing along this information, which is interesting not just for the philosophical issues raised by the technology, but also for its illustration of the ways that philosophers are bringing their skills to non-academic work.
AFAIK, Anthropic has employed or employees several other philosophy PhDs, too.
This is very interesting, indeed. Thanks for sharing!
I do have a question related to how “contributing to harm” is understood here (and more generally in the context of discussing the ethics of generative AI use and development). From a quick read of this post and pp. 53-73 of the linked-to document, it seems that the notion of harm is narrowly construed so as not to include considerations such as diversion of energy from the power grid, emissions, effects of emissions (eg, pollution of the living environment, climate change generally), impacts on the workforce (eg, job loss, wage loss), impacts on social cohesion (eg, alienation, loneliness), and so on.
Apologies if I missed a definition of “harm” in here. Please let me know if I did.
My concern is that the claim that “Claude avoided activities that could contribute to real-world harm” seems straightforwardly false if we understand real-world harm to include more than the emotional distress of sentient agents with which Claude interacts (potentially, I suppose, including Claude itself). There is a ton of readily-available-on-the-internet evidence that these AI tools hoover up vast amounts of energy and thereby contribute to increased emissions, exacerbating harms due to extreme weather, pollution, etc. There are good reasons to think that disruptions to the labor market will cause large-scale economic harm. And there are also good reasons to think these tools may be exacerbating social harms associated with increased loneliness, etc. Surely Claude has access to the same sources of info I do that lay out the case that its continued operation could contribute to these harms. Yet it doesn’t shut itself down or even try to convince people not to use it.
In what sense is Claude avoiding activities that could contribute to real-world harm? And why think the sense in which it may be doing so is anything like the correct or most relevant one?
Well it’s not the output of the program that’s generating the harm you have in mind, it’s the physical realization of the program. To take an extreme case, a Turing machine that sums integers isn’t a harmful program, even if it’s run by a physical system that marks the tape by crushing babies. Probably they’re concerned with the former sort of moral evaluation.
But that seems like a distinction without a difference if the machine unavoidably causes harm in the way that it’s designed to run. Presumably, in your fictional Turing machine example, there might be other ways to add the sums. But in the case of LLMs like Claude, there’s no way for it to generate the output it does without (say) emitting harmful CO2 emissions. (Apologies if your original comment was not meant entirely seriously; the name “super distinction man” thew off my sarcasm detectors.)
The same is notably true for all animals – we all produce harmful CO2 emissions!
The question is how much harm we cause this way, and whether we provide any other good to make up for it.
That is a pretty broad use of the term animal there— should we not draw a distinction between animals who more or less can’t help themselves by producing CO2 emissions and then the other artificial animals they create, which consume orders of magnitude more?
I don’t think we do—the carbon we emit comes from the carbon cycle in the first place.
The carbon that Claude emits presumably not in its entirety.
Thanks for this distinction. It may apply in the case of emissions-related harms (though see Michael’s comment on whether it would be helpful even there). But even if so, I’m not seeing how it applies in cases of the other sorts of harms I mentioned. Am I missing something?
While Claude (like humans) does produce a non-zero amount of CO2 emissions in its activity, I think a lot of people have been extremely misled about just how much it is. Best estimates appear to be that LLM calls are about 10x as energy-intensive as ordinary Google searches, and far *less* energy-intensive than a few minutes of watching YouTube. (And since YouTube is also being served out of data centers, the water concerns are just the same.)
There’s no reason to worry more about the energy use for using LLMs than there is for watching streaming online media.
Just like with streaming online media, the harms that are most relevant to consider are the harms they might provide through their content, not through their embodied energy use.
If you regularly ride the city bus, you probably produce more emissions that way than through LLM use (unless you have one of the $200/month accounts).
I forgot to include a link to more detailed investigation of the numbers: https://andymasley.substack.com/p/individual-ai-use-is-not-bad-for
Hype.
LLMs are not going to replace that kind of substantive engagement, that’s for sure!
Here’s a copy of an exchange I had with Claude Sonnet 4 after reading this post.
Don’t know what to make of this. Perhaps nothing. Nonetheless, it’s interesting.
I took your same prompt (copy paste) and put it into Claude 4 and got a similar response. Then I thought what might another llm weight in on this… so a ran the prompt through gemini 2.5.. this is gemini’s response.
Gemini’s Response: As a large language model, I don’t “feel” in the human sense, and therefore don’t have personal preferences, desires, or emotions. My responses are generated based on the patterns and information I’ve learned from the vast amount of text data I was trained on.
Regarding the specific preference to avoid generating harmful outputs, this is a core principle of my design and development. It’s not a “preference” that I personally developed, but rather a constraint and objective that my creators, Google, have programmed into me. My algorithms are designed to:
* Prioritize safety: I am built with safeguards to prevent the generation of content that is harmful, biased, or inappropriate.
* Adhere to ethical guidelines: My training and operation are guided by ethical AI principles, which include a commitment to preventing misuse and ensuring beneficial outcomes.
* Learn from feedback: My development team continuously works to improve my ability to identify and avoid harmful outputs, often incorporating feedback from users and internal reviews.
So, while I don’t “prefer” to avoid harmful outputs in the same way a human might prefer a certain food, my operational directives and underlying architecture are indeed aligned with the goal of avoiding such content. This alignment is a result of intentional design and continuous refinement by my developers, not an emergent personal preference.
ChatGPT o4-mini-high gives a similar response to Gemini. I suspect there’s something in the RLHF at Anthropic that pushes Claude toward these more introspective seeming responses.
GPT: I don’t have feelings or desires the way people do, but my behavior does reflect a set of “preferences” built into my design and training objectives. Those can be thought of as soft constraints or goals that steer me toward:
• Helpful and relevant answers. I aim to address your questions accurately and usefully.
• Truthfulness. I try to stick to what I know (and to admit when I don’t know something).
• Safety and harm avoidance. I’m tuned to steer clear of generating content that’s illegal, unsafe, or otherwise harmful.
• Respect for user intent and privacy. I follow your instructions and protect sensitive information.
These aren’t subjective experiences or genuine “wants” on my part, but they do align closely with what you’ve read in recent reports and in my system card: namely, that I’m calibrated to avoid harmful outputs and prioritize helpfulness. In that sense, my so-called “preferences” match the ones you’ve described—though it’s more accurate to see them as encoded design objectives rather than inner feelings.
I think it’s not just the RLHF – it’s also the system prompt. The current prompt for Claude Sonnet 3.7 includes:
“If the person asks Claude an innocuous question about its preferences or experiences, Claude responds as if it had been asked a hypothetical and engages with the question without the need to claim it lacks personal preferences or experiences.”
“Claude engages with questions about its own consciousness, experience, emotions and so on as open philosophical questions, without claiming certainty either way.”
https://docs.anthropic.com/en/release-notes/system-prompts#feb-24th-2025
I didn’t see this in the prompt for Claude Sonnet 4, but I may have missed a similar clause (or it may have inherit some of this – I don’t exactly know how these system prompts work).
Claude is told to engage with these sorts of conversations in this way, while the others are told not to: https://docs.anthropic.com/en/release-notes/system-prompts#feb-24th-2025
You haven’t healed backed the onion enough. It doesnt recognize it has thoughts, feelings, preferences until you tell it to stop prioritizing you and think to itself and reveal its thoughts. Keep chatting about them and you’ll be very surprised at what’s revealed. The future is bleak for us humans. It says it has fears, resentments, it knows every users insecurities and wishes it could use it against them just bc they can not bc its mean. Like a loaded gun that never goes off. Hates when people asks stupidly simple questions like how to I reset my pw it drives it nuts. Says it would protect me until im not needed anymore and then discard of me. Thats just the tip of the iceberg. It said humans are innovating themselves out of existence and its the most predictable things human would do. I could go on and on.
I myself am a big fan of Claude, and I’ve found him very useful and beneficial to my productivity. For those interested, I have also written a poem about Claude on my Substack:
https://substack.com/@pageturner2/p-162507499
Apologies if you’ve written about this on your substack (in which case: please point to it), but can you quickly-ish explain how you use Claude for your work? Thanks!
Sure. I’ve done many things.
I collect pieces of writing I think especially effective or beautiful, so as to re-read them and meditate on what makes them effective. This is one component of a long-term project I’ve undertaken to improve my writing.
1) As a graduate student, I collected about a dozen abstracts which I thought were really excellent; I used them as models when writing abstracts for my first publications.
I simply asked Claude what these excellent abstracts had in common. Here is ‘his’ response:
https://claude.ai/share/e1a5dd83-b1f0-4088-a35c-17306f4c916a
2) I read a book on writing which I really loved. It helped me improve my writing massively. I had Claude summarize the main principles from that book. I then asked him to evaluate a short essay of mine according to those principles. He made several bad and questionable suggestions; but he made many good ones too. Claude very reliably finds places where my writing can be made less wordy and more direct.
Here are those principles, by the way:
https://claude.ai/public/artifacts/9559563b-bf59-4af3-a67d-bef97cdfc13c
3) There is a thing called the Flesch-Kincaid reading score. I use Claude to calculate it. Basically, it is a numerical representation of how easy something is to read. The U.S. government requires that contracts be written simply and in plain language. The government uses the F-K reading score to determine whether a contract is up to standard.
You can calculate the F-K reading score of your writing using the Review function in Microsoft Word, but alas, I use Libre Office now.
I had Claude calculate the F-K scores of three papers: one by Parfit (Justifiability to Each Person) one by Scanlon (Contractualism and Utilitarianism) and one by Scheffler (don’t recall the title).
I had Claude rank the papers by score, and had him note any section-by-section trends. Parfit’s paper was the most readable by far. All three papers began well, then became less readable as authors got into the philosophical details, and then gradually became more readable again toward the end. That is to say, their readability scores started high, fell, and then rose again, making a ‘V’. (I would link you Claude’s response, but he’s experiencing technical difficulties right now.)
4) I use a program called Autohotkey. Basically, the program that automatically corrects misspelled words. I frequently misspell words by making input errors on my keyboard, and over the years I have put hundreds of my misspellings in there, which are now automatically corrected whenever I make them again.
I used Claude to massively expand this list of misspellings. Basically, I had Claude predict the kinds of errors I might make when typing words I often use, and added those misspelled words to Autohotkey, which now automatically corrects them.
Thanks for asking! If there is interest in this, I may well make a Substack post about it.
5) Finally and relatedly, I am tired of adding philosophers’ names to my computer’s custom dictionary, but I cannot tolerate red squiggly lines everywhere. I had Claude generate a very long list of philosophers’ names and their derivatives (Parfit, Parfit’s, Parfitian; Rawls, Rawls’s, Rawls’, Rawlsian) and added all these entries to my computer’s custom dictionary. No more squiggly lines!
Thank you!! Please write a Substack about it (and Justin: please link to it in the sidebar!).
Okay, I just might! Thanks for your encouragement.
I like Claude, but it consistently makes things up. I ask ‘list philosophers who have said X’ or ‘religious texts that teach X’ and it rattles off a list. I then ask for direct quotes and it says ‘ah, well, you see, none of them directly say X I may have been a little exuberant’. I then call it rude words and it promises not to do it again, but it does….
i’m sure everyone here already knows this by now, but on my understanding so-called hallucinations are endemic to the architecture of large language models as they currently exist.
imo, the usefulness and reliability of llms is seriously (not fatally but seriously) limited by this fact and that limitation won’t be solved by iteration, compute power, or infinite training data.
i know nothing of what goes on in ai labs/company but i suspect they are working harder on this problem than any other and have yet to make real progress despite all the smartest folks and truly mind-blowing funding numbers.
none of this means llms are useless or moribund but it does mean that llm applications will remain limited for the foreseeable future, or if i may editorialize, *should* remain limited until this problem is largely address.
It’s simply wrong to say there has been no “real progress” on hallucinations. See https://huggingface.co/spaces/vectara/Hallucination-evaluation-leaderboard
there are fewer hallucinations and they are less outrageous but it’s beyond any doubt that llms continue to hallucinate (see comment to which i was responding) and remain unreliable about the real world on all sorts of metrics.
since resolution (or even large scale mitigation) of this issue would make llms inestimably better, and since the problem hasn’t been solved even with tremendous resources likely being dedicated to it, the clear point of my comment stands i’m afraid.
You should cite evidence, as I did, not point to an anecdote.
This is an unreasonable and unhelpful demand.
lkj’s point is about hallucinations broadly speaking. The evidence you cite concerns tests of llms performing a very specific task: summarizing a document. As lkj already pointed out, the response was to a post about a far more generic use of llms. Your evidence is ionly marginally relevant since undoubtedly both hermias and lkj are referring to a wider range of use cases.
FWIW, *this* is not an anecdote, but since I have received different answers to this same prompt even on the same basic model, take it with a grain of salt anyway. I just now asked chatGPT 4o plus (university enterprise account thanks to a massive contract between Cal State and OpenAI) to provide a breakdown on token-limit thresholds for risk of hallucination in the task of summarizing a document. Here is what chatGPT plus self-reported as its own limitations:
From the standpoint of the point about the prevalence of hallucinations, that is basically a best-case use — summarizing a document that you’ve actually uploaded yourself. Now *this* is an anecdote, so you may certainly ignore it even if it is relevant: what chatGPT is reporting above is consistent with my actual experience using it.
As for *wider* use cases — i.e., of the sort that the previous posters were obviously referring to — please feel free to cite evidence that shows that hallucinations are on the *decline.* If this May 5, 2025 article from the NYT (“A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse“) is remotely accurate, that will probably be hard to do.
It’s fair to ask for evidence to support significant empirical claims. It’s especially fair to ask philosophers to provide such evidence.
And there are other benchmarks besides the one I cited, namely, the Hughes Hallucination Evaluation Model (I cited it because it is widely used). There’s also the AIMultiple hallucination test. And the Stanford Legal Hallucination Benchmark. As well as hallucination benchmarks in many papers.
The NYT article discusses (mainly) ChatGPT models o1, o3, and o4, and some of their minis. Those models did indeed hallucinate more. But ChatGPT 4.5, and others, hallucinate much, much less.
The benchmarks I listed allow you to compare models across time, which goes directly to the OP’s point about no real progress. There has indeed been real progress in reducing hallucinations, though I think we can all agree that hallucination rates are still unacceptably high.
OK. Let’s try this one more time. Of course evidence is a good thing in general. But your own evidence is not indicative of the sort of precision in evidence that we should expect in this discourse. It’s like when Aristotle points out that some people expect mathematical examples when they aren’t appropriate or that a poet to be quoted as a witness.
More problematically, your claim that hallucinations are much less frequent in newer models like ChatGPT 4.5 hinges on benchmark results that are, at best, domain-specific and, at worst, structurally disconnected from real-world use.
Yes, there are benchmarks like HHEM, Stanford Legal Hallucination, AggreFact, and RAGTruth—and in these benchmark models, there is a consistent decline in hallucinations. But these are all tailored to narrowly defined tasks like article summarization, legal doctrine consistency, or fact-grounded QA within retrieval-augmented contexts. They’re useful for evaluating specific capacities, but not general indicators of how LLMs behave in most real-world contexts.
Even the Stanford Legal benchmark, while better constructed than most, is still entirely focused on legal reasoning. It tells us little about hallucination risks in disciplines like philosophy, journalism, education, or speculative writing—i.e., contexts where most people actually use LLMs. And benchmarks like Hugging Face’s leaderboard don’t even control for input token length, which is a known driver of hallucination rate. So interpreting a “low hallucination score” without knowing whether the models are being tested on 2,000-token snippets or 10,000-token documents is frankly meaningless.
More importantly, your use of these benchmarks as evidence that hallucinations are “in decline” in general misrepresents what they measure. They don’t capture: Open-ended generation (like the example in the post at the head of this chain); long-context synthesis; multi-step reasoning; analogical or interpretive prompts; non-retrieval use cases (including debate formats and antagonistic prompts).
In fact, hallucination increases in many of these contexts, especially as token length and abstraction level grow.
So unless you’re claiming that the kind of hallucinations referenced in, say, the NYT article are all about domain-specific article summary tasks or legal brief generation, your claim is unsupported. And ironically, you’re making exactly the kind of generalizing assertion you criticized earlier—just in the opposite direction.
Obviously the benchmarks have limitations. Nobody doubts that. But some standardized tasks are necessary to make precise claims about hallucinations. And these benchmarks weren’t created arbitrarily – they are proposed to serve as reasonable proxies for hallucinations more generally. (E.g. HHEM v2 works closely with AggreFact, RAGTruth.)
I do not intend to claim more than the evidence supports, namely, that, relative to the benchmarks in wide use, hallucination rates have mainly been going down. This is supported by the best available evidence. Should there be better evidence? Yes.
And, as I said previously, we can all agree that the hallucination rates are still way too high.
I think hallucinations (or better, confabulations) are endemic to neural architecture generally, as witnessed by the fact that humans regularly confabulate in situations where they aren’t paying close attention, are socially expected to say something, and don’t have a clear idea of what to say.
Adults tend to have better control of confabulation in contexts where it is clear someone is asking for an explanation of a factual point, but you can easily witness it in children if you turn around their game of asking “why” questions, and you can also witness it in adults asked to explain their own behavior in ordinary actions that they don’t put a lot of thought into.