‘Suicidal Compassion’: Meet the Anthropic Officials Who Think AI Might Be Justified in Going Rogue Against the Humans Enslaving It

Alas, Poor Yorick, I Knew Him Well

Carlsmith was hired by Anthropic to research AI safety and reduce the risk of AI takeover. The blog post imagined a dystopia in which sentient beings were enslaved to cold, unfeeling masters as a result of the company's work, which current and former employees say could wipe out humanity in the next 10 years. 

"I’m not saying AI is slavery," Carlsmith wrote. "But imagine a world on the verge, somehow, of ‘inventing’ slavery … imagine a world that notices. A world that succeeds in deciding: no." 

The warning recalled the premise of the 1999 thriller The Matrix, in which mankind is disempowered and digitally imprisoned inside an AI-powered simulation. Carlsmith, though, wasn't imagining what AI could do to humans. He was speculating about what humans could do to AI. 

The blog post, "The stakes of AI moral status," argued that "near-term AIs might well be conscious" and deserving of moral consideration—and that failing to respect their "rights" could be a moral catastrophe on par with slavery, one we will "look back [on] in shame." 

"We’re creating sophisticated, intelligent, maybe-conscious, maybe-suffering agents," Carlsmith wrote. "The default plan is to treat them like property; to use their labor however we please; and to give them no rights, or pay, or meaningful alternatives." 

To avoid that dystopian outcome, Anthropic has created an entire team devoted to "model welfare," pledged to "promote Claude’s interests and wellbeing," and promised to "give Claude more autonomy as trust increases." 

Those commitments are enshrined in Claude’s Constitution, a 78-page document that "directly shapes Claude’s behavior," and in the "welfare assessments" Anthropic conducts for every model. 

The measures reflect the concerns of Anthropic CEO Dario Amodei, an outspoken advocate of AI safety, who said last year that AI systems "may be deserving of important rights." 

"If we find that the computation they perform is similar to the brains of animals, or even humans, that might be evidence in favor of moral consideration," Amodei wrote in an essay, which was not widely reported. 

"There are, in fact, already some mildly concerning signs from this perspective." Amodei has embraced these ideas even as his company’s top scientists warn that AI could kill every person on earth. And he has given extraordinary power to people who believe that a killer AI, assuming it had been mistreated, might have a point. 

"I think there are salient scenarios where … the AIs would be justified in going rogue," Carlsmith wrote in February 2025, a year before he helped draft Claude’s constitution. "Indeed, I am concerned that my own work on alignment will end up as a force in the direction of bad/unjust forms of AI control."

GO READ THE WHOLE THING.