Anthropic warns AI could soon improve itself without human input and calls on major labs to agree on a coordinated, verifiable plan to slow or pause frontier AI development before society loses control.
Image Source: X
Anthropic said on Thursday that frontier AI developers should establish a coordinated, verifiable way to slow down or temporarily pause development if advanced systems begin improving themselves faster than society can manage the risks.
The proposal came in a blog post written by Marina Favaro and Jack Clark. Favaro leads the Anthropic Institute, the company’s internal research organisation. Clark is Anthropic’s co-founder and head of policy.
Advertisement
The post was titled “When AI Builds Itself.”
Advertisement
The authors wrote: “We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology.”
At the centre of Anthropic’s concerns is the idea of “recursive self-improvement.” This is the theoretical moment when an AI system becomes capable of rewriting its own code, fixing its own weaknesses, and upgrading its own abilities without direct human oversight.
The two authors said model advances appear to be getting closer to this concept, which refers to AI systems that are capable of independently improving themselves and expanding their own capabilities simply by writing their own code.
They acknowledge that this has not happened yet and is not necessarily inevitable, but they warn it could happen much sooner than anyone is prepared for.
Jack Clark told BBC News that reaching fully self-written code could be possible within two years.
The company disclosed internal data about accelerating model capabilities. Using public benchmarks and previously unreported data from within Anthropic, the Anthropic Institute showed that AI is already accelerating the development of AI systems. Anthropic engineers on average ship eight times as much code per quarter as they did from 2021 to 2025.
As a further example, Anthropic said that as of May, more than 80 percent of the code merged into its codebase was authored by Claude.
The company said such advances could leave governments, regulators, and even AI developers struggling to understand and assess increasingly capable systems as the technology evolves.
Anthropic cautioned that unilateral or poorly coordinated slowdowns could backfire if less cautious actors continue advancing, potentially reducing overall safety.
“A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions,” Favaro and Clark wrote.
A unilateral pause by a single company would be easier to implement, Anthropic added, but would have limited impact, primarily shifting leadership rather than fostering broader global deliberation.
The company also said any pause framework would need to include rules on what conditions would trigger or lift it, and clarity on who would oversee compliance.
Anthropic conceded that it would likely require something similar to the Cold War-era treaties that slowed down the pace of nuclear weapons proliferation, but noted that it is a lot easier to mask AI training runs than it is to conceal missile silos.
Unless some kind of ironclad verification regime is put in place, with cooperation from countries such as China, any pause would risk creating a scenario where competing nations simply ignore the pause and take the lead in AI development.
“Tracking decentralized computing resources, private data centers and algorithmic research globally is far more difficult than monitoring something physical, like nuclear facilities,” one analyst noted.
Anthropic’s research arm, the Anthropic Institute, plans to study and help build systems that would be necessary to support a slowdown.
Anthropic plans to organise conversations in the coming months with policymakers, researchers, and others to help answer questions around recursive self-improvement and a verification system.
The post stated: “The window to investigate the questions together is here, and people outside AI companies should be involved in this deliberation.”
The Anthropic Institute started in March 2026 as a think tank studying AI, led by Jack Clark.
The proposal has already drawn scepticism. David Sacks, a venture capital investor and informal adviser to President Trump, has accused Anthropic’s leaders of running a “regulatory capture agenda.”
On a recent podcast, Sacks said the regulatory capture agenda in Washington could lead to an effort to ban open-source models, which are versions of AI systems that are far cheaper for organisations to use and develop internally.
Others have suggested that Anthropic’s warnings about the dangerous potential of its own tools could also be a marketing move.
The call for a pause comes as Anthropic itself is expanding rapidly. Anthropic’s run rate is on track to reach $50 billion in annualised revenue by the end of this month, up from $9 billion at the end of 2025.
Anthropic recently concluded a fundraising round that valued the company at almost $1 trillion and filed confidential paperwork to begin the process of publicly listing its shares.
The company has recently emerged as the front-runner in a competition for AI supremacy with ChatGPT-maker OpenAI. The latter is also likely to file paperwork for an initial public offering soon.
Advertisement
