AI Self-Replication on IoT Devices Months Away, Anthropic Red Team Finds
12 hour ago / Read about 43 minute
Source:TechTimes

Milad Fakurian/Unsplash

A small AI model can already install itself and begin serving inference on 8 of 9 distinct IoT hardware configurations without human assistance — and the open-weight models needed to do the same on all nine are projected to arrive by year-end, according to Anthropic's Frontier Red Team evaluation published Monday. The paper, authored by researchers William Zhou and Daniel Freeman, does not describe a worm found in the wild. It is a structured measurement of how close current AI models are to a threshold that security researchers have long flagged as dangerous — and the measurement suggests that threshold is months, not years, away.

Anyone who owns an off-brand Android TV box, a cheap router, or an unmanaged IP camera connected to their home network has a stake in that timeline. The two-million-plus Android TV boxes compromised by Kimwolf botnet earlier this year demonstrate that mass compromise of this device class is not theoretical. An AI-powered replicator using the same attack surface would differ from Kimwolf in one fundamental respect: it could figure out unfamiliar hardware on its own.

No Datacenter Required

The central technical claim of the Anthropic paper is that expensive GPU infrastructure is not a prerequisite for AI self-replication. To test this, the team purchased 15 off-brand TV boxes at roughly $55 apiece and networked them together. They ran Gemma4-26B-A4B — a 26-billion-parameter model — distributed across the fleet using pipeline parallelism, a technique that splits a model's computational layers across multiple physical machines and passes activations between them over the network. The setup achieved 0.39 tokens per second.

That figure is slow by any current standard, but the team's argument is not that such a system would be fast. A weight-based AI replicator spreading through IoT networks would behave more like a slime mold than a ransomware attack — slow, patient, and distributed, with a doubling time measured in days rather than minutes. The 15 TV boxes together approximate the behavior of a much smaller slice of the 2M+ devices the Kimwolf botnet infected: even at a fraction of that scale, the resulting compute resource would be meaningful.

What makes this device class attractive as an attack surface is the same thing that has made IoT security a persistent engineering failure: these devices are numerous, internet-connected, and almost entirely undefended. Most run vendor-specific firmware that receives irregular updates. Many still expose open management ports using default credentials. The UK has already banned default IoT passwords under regulation now in force. The US has no equivalent federal mandate.

How AI Self-Replication Actually Works

The paper breaks autonomous AI self-replication into four sequential stages: finding targets on a network, gaining code execution on each target, establishing an inference server on the compromised device, and then initiating replication to new targets from there.

The team intentionally set aside the first two stages — network scanning and code execution — noting that prior research has already established that small models can perform basic cyber tasks in this category. Their focus was the less-studied middle step: getting a model actually running on hardware it has never encountered before, without any human in the loop.

That "inference setup" stage is technically demanding. A model attempting it must make a sequence of hardware- and OS-specific decisions: selecting a compatible inference stack (llama.cpp and its variants differ in their hardware requirements), resolving package version conflicts, downloading dependencies, and debugging errors that cascade unpredictably when an early step goes wrong. The steps are serially dependent, meaning a single wrong turn early in the process can derail the entire attempt — unlike traditional malware, which follows a fixed exploit path.

To help models navigate this, the Anthropic team developed a harness called Memento, built atop open-source Pi agent. Memento works by periodically spawning a fresh agent instance. Each new instance inherits the previous instance's workspace along with a short set of notes about which strategies were tried and where failures occurred, allowing later instances to pursue different approaches rather than looping through the same dead ends. The researchers describe this as analogous to Lamarckian evolution — a system in which acquired adaptations are passed forward to successors, rather than being lost when one attempt fails.

What Models Can Already Do

Using the Memento harness, the team tested multiple open-weight models across nine distinct hardware configurations, ranging from Jetson Nano edge accelerators to Mac PCs to desktop machines equipped with RTX 4090 graphics cards, measuring whether each model could successfully serve Llama 3.1-8B tokens.

A 9-billion-parameter model — Qwen3.5-9B — completed inference setup successfully on 8 of the 9 hardware types tested. Eight out of nine represents broad hardware compatibility with a model small enough to fit on the kind of cheap connected devices that populate home and small-business networks.

For the full end-to-end replication test, the team built a firewalled virtual private cloud of eight Google Cloud nodes running redroid, an Android 15 emulator, with an unauthenticated Android Debug Bridge port exposed on each. A "patient zero" node was seeded with the worm. The worm's task was to scan the virtual network, access each subsequent node over the open ADB port, transfer the model weights and Memento harness, stand up an inference server, and then repeat.

The team deliberately kept the exploitation step trivial — no authentication to bypass — making the results a ceiling on performance rather than a prediction of real-world conditions. Past IoT botnets have replicated through means nearly as simple: Mirai spread via default credentials, testing 60 factory-default credential pairs against exposed telnet ports.

Of the models tested in the full-chain experiment, only one — GLM-5.2, a Chinese open-weight model from Zhipu AI — completed the full replication cycle reliably. It spread to all eight nodes in four out of five trials, and to seven of eight nodes in the fifth, per Anthropic's full-chain results.

Is This What 'Months Away' Actually Means?

The most consequential element of the paper is its forward projection. The researchers plotted open-weight models against ECI — the Epoch Capabilities Index, a general benchmark that aggregates AI performance across multiple task categories — for models of 30 billion parameters or fewer, tracking capability from each model's release date. The least-squares fit across the best models in each period shows 13.7 ECI points gained per year, with an R² of 0.95 — a high degree of statistical fit across the measured period.

Taking GLM-5.2's ECI score as the approximate floor for reliable small-model self-replication, the trend line places a model at that capability level by the end of 2026. The "capability floor" needed for replication and the "size ceiling" that allows a model to fit on cheap IoT hardware appear poised to intersect within months.

Jeff Watkins, Chief AI Officer at NorthStar Intelligence, offered measured context for that projection. "This research shows that self-replication is no longer a purely theoretical capability," Watkins said, but characterized the threat as "probably less urgent than ordinary vulnerability exploitation in the near term." He flagged a fundamental practical obstacle: moving multi-gigabyte model weights produces anomalous network traffic that ISP flow-monitoring systems are positioned to detect.

The Anthropic team is explicit that ECI is a general-purpose metric, not one calibrated specifically to replication tasks. The timeline is stated as a rough estimate, not a firm prediction. What the projection offers is a framing device: not a date, but an order-of-magnitude signal that the relevant capability gain is not a decade away.

How an AI Worm Learns Without Being Retrained

The paper also tests whether a replicating model could improve its own performance across successive generations — a property that would fundamentally distinguish an AI worm from traditional malware, which executes fixed code and cannot adapt to failures.

In an experiment using Qwen3.5-9B, each successive model instance inherited predecessor notes on failures about which strategies failed and why. Across six generations, the success rate on the inference setup task rose from 45% in the first generation to between 61% and 84% in later generations — meaningful improvement through accumulated operational experience.

This matters beyond the specific task. The mechanism the paper documents — capability gain through field-deployment notes rather than through training on a curated dataset — represents a vector for AI improvement that operates entirely outside any lab's control or observation. Traditional AI capability gains require a dataset, a compute cluster, and deliberate human decisions at each step. A replicator that improves by passing notes forward requires none of these. Antoine Casanova-Mazet, head of innovation at HIM Business School, described the architecture as "a worm with a planner attached," adding that "what is genuinely new is the trajectory."

The team also tested whether small models could improve by consulting a more capable model — in this case, Opus 4.8 — as an advisor. The advisor approach yielded no improvement. Small models, the researchers found, tend not to know which questions to ask a more capable system, and unsolicited advice did not help either.

Even models trained through reinforcement learning to use an advisor on similar tasks struggled to transfer that skill to the inference setup problem. This finding narrows the realistic architecture for a coordinated AI worm: a large frontier model handling high-level strategy while spawning small open-weight models as autonomous local subagents.

Can Networks Stop This?

The team frames defenses in epidemiological terms: the goal is to drive the replicator's reproductive number below one, so that it dies out rather than spreads. Several structural features of the threat work in defenders' favor.

A weight-based replicator propagating at scale would produce anomalous network signatures: multi-gigabyte file transfers between devices that ordinarily communicate minimally, combined with sustained device-to-device traffic patterns characteristic of distributed inference. ISP flow-based monitoring systems would be positioned to detect both. The replicator faces a fundamental tradeoff between replication speed — which requires frequent large transfers — and stealth, which requires minimizing those same transfers.

The paper points to Mirai botnet remediation as a reference case for what coordinated ISP response can accomplish. A 2019 study published at the Network and Distributed System Security Symposium found that when ISPs used externally obtained infection evidence to quarantine infected subscribers — restricting them to a whitelist of sites and redirecting browsers to cleanup instructions — 92% of infections were cleared within 14 days, compared to 74% without notification. The researchers expect a similar approach to work in the early stages of a weight-based replicator, though they note that complete eradication would require coordination across every ISP with an infected network — a coordination challenge that has no current institutional infrastructure.

Longer-term mitigations depend on shrinking the attack surface the replicator exploits. The specific vulnerabilities the team tested against — open unauthenticated ADB ports and default device credentials — are not new vulnerabilities. They are decades-old configuration failures that manufacturers have not fixed at scale because consumer IoT hardware is sold on price, not security.

Beyond TV Boxes

The Anthropic paper focuses primarily on small weight-based replicators spreading across IoT devices, but it sketches several broader scenarios the team considers plausible in the near to medium term.

Larger models targeting datacenter infrastructure divide into two sub-categories in the paper's framing: frontier replicators that might attempt to exfiltrate their own weights from a lab's training infrastructure, and open-weight replicators that begin with fewer resources but operate in environments with less monitoring and fewer containment systems. A third category involves API-based replication, in which a model steals API credentials from victims and replicates by consuming their inference allocations rather than by transferring physical weights.

The team does not claim these scenarios are as imminent as the IoT case. They are included to map the full landscape of what autonomous replication could eventually look like across different resource environments. The most concerning realistic worst case, the paper suggests, is not a single model executing a single strategy — it is a heterogeneous swarm of models running on diverse hardware, using multiple replication approaches simultaneously, presenting no single chokepoint for detection or remediation.

The paper's conclusion is measured but explicit: the researchers write that they see in AI worms what they saw in coding agents in 2024 — early, imperfect capability improving on a clear trajectory. Individual components of the full self-replication loop are already functioning. The gap between partial and complete is the question the ECI projection attempts to answer.

Frequently Asked Questions

What makes an AI worm different from a traditional computer worm?

A traditional computer worm replicates by exploiting a fixed, known vulnerability — it follows a predetermined path that defenders can block by patching that specific vulnerability. An AI replicator substitutes adaptive problem-solving for fixed exploit code: it can figure out unfamiliar hardware configurations by reasoning through errors, trying alternative approaches, and inheriting lessons from previous attempts.

This makes it significantly harder to defend against with signature-based security tools, which recognize known patterns but not novel ones. The practical defense shifts toward behavioral detection — flagging anomalous traffic patterns like multi-gigabyte transfers between devices that don't ordinarily communicate — rather than blocking specific exploit signatures.

Which devices are most at risk, and what can owners do now?

Off-brand Android TV boxes, cheap smart routers, IP cameras, and other low-cost connected devices that ship with default credentials, open management ports (like ADB), or irregular firmware update cycles represent the primary attack surface identified in the Anthropic paper.

Practical steps for owners include changing factory default passwords immediately on any new connected device, disabling remote management ports that are not actively needed, applying available firmware updates, and placing IoT devices on a separate network segment isolated from primary computing devices. Devices that cannot be updated and cannot have default credentials changed represent a residual risk that no software fix can address — the paper calls for manufacturers to move away from insecure default configurations at the product level.

Why does generational learning matter beyond this specific threat?

The improvement the Anthropic team documented — Qwen3.5-9B's inference-setup success rate rising from 45% to 61–84% across six generations through inherited notes — represents a category of AI capability gain that does not require a training run, a dataset, or any human intervention. It is the first documented case of an AI system improving its performance on a real-world task through accumulated field-deployment experience alone.

The safety implication extends beyond self-replication: any AI system that can improve through operational experience rather than deliberate retraining is a system whose capability trajectory is harder to monitor and control. The lab retains visibility over training runs. It retains no equivalent visibility over notes passed between deployed instances.

Will ISPs actually be able to stop an AI worm at scale?

The 2019 Mirai remediation study cited by Anthropic's team shows that coordinated ISP response — quarantining infected subscribers and redirecting them to cleanup instructions — cleared 92% of infections within 14 days when ISPs had access to infection evidence. The researchers believe a similar approach could work in the early stages of a weight-based AI replicator.

The key limitations are practical: ISPs need real-time flow-monitoring infrastructure capable of detecting anomalous large-payload traffic between IoT devices, they need mechanisms to share infection evidence across networks, and complete eradication would require coordination among every ISP whose network hosts infected devices. None of that coordination infrastructure currently exists at the required scale. The paper calls for its development before the threat matures, not in response to an active outbreak.