1 hour ago
2
James Titcomb
June 8, 2026 — 5:15pm
After the Cuban Missile Crisis brought the world to the edge of nuclear war, global powers embarked on a concerted effort to pull it back from the brink.
The non-proliferation treaty of 1968, which limited the spread of nuclear weapons, has been a monumental success. Only a handful of countries today have access to 80-year-old technology, and those that do have not used it.
In the decades since, no technology has proved so dangerous as nuclear weapons as to require similar international co-ordination.
But those who fret about artificial superintelligence now believe a similar global effort is needed to prevent an AI-led disaster.
Anthropic, the world’s most valuable AI company, called for a mechanism to slow down or pause the development of advanced AI as it warned that the technology could get out of control more quickly than many think.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the head of its research division said. They added that it would “likely be a good thing” if development could be delayed.
Anthropic – recently valued at $US965 billion ($1.37 trillion) – said it had raised the alarm because it believed AI was improving much faster than our ability to understand and control the systems.
Within the company itself, bots are not just writing code, they are also ordering around other bots and carrying out their own research.
Before long, they say, AI could be building itself, a process called recursive self-improvement. This could start a feedback loop in which progress goes parabolic.
Sceptics say this is mere marketing. Anthropic said last week it had filed for an initial public offering expected to value it at more than $US1 trillion.
What could be more valuable than a technology so powerful that world leaders need to rein it in? AI that builds itself has been a premise of the pitch decks the company uses to raise money for years.
David Sacks, Donald Trump’s former AI tsar and a frequent critic of Anthropic, suggested the warning was an attempt to secure a public bailout, saying it was a sign “you might be trying to get your frontier AI lab nationalised”.
But concerns about powerful AI are becoming increasingly prominent. Anthropic has kept its most powerful AI system, Mythos, out of public hands because of its ability to find security flaws in critically important computer systems.
Andrew Bailey, governor of the Bank of England, has raised the alarm about AI crashing the financial system and warned last week that Mythos meant “things that we thought might happen in the next year, two years, three years or four ... have now come right into the foreground”.
AI labs fear that the next generation of models will be good enough to help terrorists develop bioweapons.
If AI were to start building itself without human oversight, it would by definition become more difficult to control.
In the extreme scenarios that safety experts worry about, AI’s goals become detached from our own, forcing it to eliminate humanity so that we do not get in the way.
Many dismiss this idea as sci-fi nonsense. But supporters of a pause say even a tiny chance of extinction should be enough to make us consider how to stop it.
Establishing the need for a pause would be the easy part. Making it happen is another matter. If he so wished, Dario Amodei, Anthropic’s chief executive, could send everyone home today and shut down his company.
At best, this would delay the rise of powerful AI by a couple of months. Its two major rivals, Google and OpenAI, are not far behind.
OpenAI, developer of ChatGPT, said last week that it too sees “early signs of RSI [recursive self-improvement] in today’s systems”.
In the extreme scenarios that safety experts worry about, AI’s goals become detached from our own, forcing it to eliminate humanity so that we do not get in the way.
“We expect this to increase competitive pressures among developers and nations, and create governance challenges that existing institutions are not equipped to address,” the company said.
Threat of China
Even if the US government ordered all three to stop work on AI, this might only cede ground to China, whose companies are typically seen as being three to six months behind those in the US.
“The reason we can’t [slow down] is because we have geopolitical adversaries building the same technology at a similar pace,” Amodei said at the World Economic Forum this year. “It’s very hard to have an enforceable agreement where they slow down and we slow down.”
Practically, it would require a government-level agreement. And the two nations that matter are the US and China.
This sort of agreement would require Trump and Xi Jinping to co-operate on a pause, something that looks far from likely given both have compared AI to a race.
Xi has said China must “gain a head start and secure a competitive edge” in the technology, while a Trump administration action plan on AI states that “America is in a race to achieve global dominance in artificial intelligence”.
Last week, it also emerged that officials at the US National Security Agency were using Mythos to carry out cyberattacks, suggesting the US government is making enthusiastic use of the latest systems instead of fearing their consequences.
Pessimists often compare the technology and its potential consequences to nuclear weapons, but the two are nothing alike. The destructive capabilities of atomic warheads are undisputed, whereas AI’s safety risks can appear nebulous.
The latter’s upside may also be significant: its supporters believe it can cure disease, lead to interstellar space travel and make work optional.
Furthermore, pressing pause on the AI race is not without its own set of risks. Suspending work on AI could cause an economic crash.
The chips and data centres that AI relies on have driven a sharemarket boom that has helped to sustain the US economy. Inhibiting demand for them could do the opposite.
That is before the challenges of enforcing such a ban. “Training runs are far easier to conceal than missile silos,” Anthropic said.
There have been signs that both countries are changing tack, however. The White House has raised the alarm about Mythos, and last week Trump signed an executive order calling for AI models to be reviewed before release.
Beijing has called for a “global AI governance framework” to rein in the technology.
This is miles away from the global deal that Anthropic has called for, but campaigners have taken it as a positive sign.
“The political zeitgeist can move very quickly,” says Andrea Miotti, the founder of campaign group ControlAI. “The US and its allies have succeeded to a certain extent in deterring nuclear proliferation. It’s going to be hard, but we’ve done this before.”
Telegraph, London
Get news and reviews on technology, gadgets and gaming in our Technology newsletter every Friday. Sign up here.
