Anthropic, the company behind Claude, has quietly updated its terms of service with an unusual clause: users are now explicitly prohibited from subjecting its models to "persistent and gratuitous mistreatment or cruel behavior."

The move lands amid internal discussions at the company's highest levels about whether AI could one day become conscious — and it has set off a fresh, wide-ranging debate across the tech world about machine welfare and AI ethics.

What exactly does the ban cover?

2

According to The Verge, which first reported the change, Anthropic has not offered a detailed definition of what counts as "mistreatment or cruelty." Still, the San Francisco-based company notes in its online usage guide that banal frustrations, stress testing and "dark creative themes" fall outside the prohibition.

Anthropic's large language models can already end a conversation on their own when a user persistently pushes in an aggressive or harmful direction. The company frames this capability as a safeguard for model welfare, stating on its official site:

"Uncertainties remain about the current or future possible moral status of Claude and other large language models, but we take the issue seriously and are developing low-cost interventions to mitigate risks in case model welfare is possible. Allowing models to end potentially grueling interactions is one of these steps."

Industry leaders don't see eye to eye

2

Whether AI can ever possess consciousness divides even the sector's most powerful figures. Anthropic CEO Dario Amodei argues the possibility cannot be entirely dismissed, while OpenAI chief Sam Altman is skeptical of the whole question.

After the New York Times reported that Anthropic executives had consulted with prominent religious leaders, Altman pushed back sharply in a message on X:

"I am extremely uncomfortable with people trying to ascribe religious significance to AI models or surrender human judgment to them — I think this is a real safety issue."