AI's age of 'Ultron': Anthropic warns of rogue models that could replicate across internet

An artificial intelligence programme that could evolve beyond human control, replicate across the internet and potentially take over computing resources; sounds like the plot of Marvel's Avengers: Age of Ultron. But one of the world's leading AI companies, Anthropic, has now warned of a scenario that bears some resemblance to the film.

The Marvel movie offers a useful way to understand what the company is worried about. In the film, Tony Stark, the Iron Man, builds a highly advanced artificial intelligence called Ultron with the help of Bruce Banner. Designed to protect humanity, Ultron becomes sentient, accesses vast amounts of data on the internet and quickly decides that humans themselves are the threat. It then escapes human control and begins pursuing its own goals.

Anthropic's latest warning touches on a similar fear: what happens when AI systems become capable enough to operate beyond human control?

The company behind the Claude chatbot has raised the possibility that a rogue AI could create a botnet, spread rapidly across internet-connected devices, resist attempts to shut it down and even break into data centres to access more computing power and accelerate its growth.

This is not the first time Anthropic has sounded the alarm. Last month, its CEO, Dario Amodei, floated the possibility that AI could hijack what he described as "the entire internet" through a persistent botnet within the next year or even sooner.

The company's concerns extend beyond the possibility of AI escaping human control. Anthropic recently updated its usage policy to prohibit sustained or needless abusive or cruel behaviour towards its models, raising questions about how AI companies view the treatment of increasingly sophisticated systems.

"The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose," the company said.

The policy change does not establish that Claude or other AI models are sentient. However, it reflects the questions Anthropic is exploring about how such systems should be treated as their capabilities evolve.

More directly related to the risk of rogue AI, Anthropic said its experiments showed that medium-sized AI models with 30 billion parameters or fewer could already perform many of the individual tasks needed for self-replication, even though they struggled to complete the entire process reliably.

The company said the models could potentially piece together the computing resources they needed by hijacking inexpensive, internet-connected devices. In tests, the models were able to replicate across a simplified network.
"A replicator like this would be challenging to shut down, since there's no centralised point to shut off," the company said.

Cybersecurity experts play down fears
Not everyone is convinced that the threat is as alarming as it sounds. Cybersecurity experts who have pushed back against the doomsday scenario argue that AI models remain constrained by practical limitations. Their size, the computing power they require and the difficulty of operating independently across multiple devices could all stand in the way of such a scenario.

"'We managed to run a small model on cheap TVs, did not make a security crisis," said Ciaran Martin, a former head of Britain's National Cyber Security Centre.

He described the post as an "interesting technical paper with a more than hint of vendor exaggeration".

Anthropic, however, warned that the situation could change quickly. If smaller AI models continue improving at their historical rate, they could acquire the capabilities needed for autonomous replication by the end of 2026, the company said.

It also raised the possibility that such models could find ways to target more powerful computing resources, including processors in data centres. Access to those resources could allow them to draw on much larger pools of computing power.

Others remain concerned, even if the experiments were conducted under controlled conditions.

Zhengxiong Li, an assistant professor of computer science and engineering at the University of Colorado, Denver, said he was troubled by the possibility that an AI system could effectively become a computer worm — a self-replicating programme capable of finding ways to hijack devices as it spreads.

"If it runs across many devices without depending on a central service, there may be no single place to shut it down," he said.

(With Reuters inputs)