Anthropic Co-founder and CEO Dario Amodei speaks at the "How AI Will Transform Business in the Next 18 Months" panel during INBOUND 2025 Powered by HubSpot at Moscone Center on September 04, 2025 in San Francisco, California. Chance Yeh/Getty Images for HubSpot

Three days after Anthropic CEO Dario Amodei warned of AI agent swarms that could seize large parts of the internet within six to twelve months, a new academic study explains one of the core reasons why: the safety monitoring infrastructure that enterprises rely on to catch AI agent compromises is not watching the right thing. Attacks can succeed at the planning, memory, or tool-invocation layers of a deployed agent — layers most organizations do not monitor — while the final response the system produces looks perfectly clean to every safety filter currently in common use.

The paper, "SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration," was submitted to arXiv on September 11 by researchers Md Jueal Mia, Yanzhao Wu, Selcuk Uluagac, and M. Hadi Amini. As a Systematization of Knowledge (SoK) paper — a genre designed to synthesize and reframe accumulated research in a field — it draws on the full body of jailbreak attack and defense literature and applies it to the agentic pipeline attack surface. What it finds is that three assumptions underpinning current AI security practice are wrong, and that the wrongness of all three points in the same direction: toward a monitoring gap that enterprises are operating inside right now, without knowing it.

What's at Stake If Your Agent Is Deployed Right Now

If your organization is running AI agents in production — systems that browse the web, read documents, invoke APIs, execute code, or coordinate with other AI models — the central implication of this research is direct: you cannot tell from your final-output safety metrics whether your pipeline is secure. A low attack success rate on output monitoring tells you the final responses passing through your safety filter look clean. It tells you nothing about whether the planning step was manipulated, whether hostile instructions were written into the agent's memory store, or whether the tool invocations the agent made on its way to producing that clean response were doing something they should not have been.

The scale of exposure is not hypothetical. A survey published the same week found that 77% of enterprises lack verified inventories — only 44% run active discovery tooling — and 74% trust their testing to catch failures while just 19% have an automated gate to block bad releases. That confidence gap — between what organizations believe their monitoring catches and what it actually catches — is exactly the gap the Mia et al. paper is documenting at the technical level.

Finding One: Aligned Does Not Mean Defended

The first and most counterintuitive finding is that a model's native safety alignment does not predict its robustness against adversarial jailbreak attacks. These are, empirically, orthogonal properties.

The distinction requires understanding what alignment training actually optimizes for. Techniques like RLHF and Constitutional AI train a model by presenting it with prompt-response pairs and applying a reward signal based on human preference scores. The model learns a pattern: in contexts that look like this, produce responses that score highly. That pattern is what produces the reliable, helpful, appropriately cautious behavior that well-aligned models exhibit in everyday use. It is not, however, a mechanism for detecting adversarial inputs — inputs that have been specifically crafted to exploit the boundaries and failure modes of those learned patterns. The model that refuses a direct harmful request in 99 out of 100 cases can still be reliably defeated by a well-engineered adversarial prompt that frames the same request in a way that does not trigger its learned refusal behavior.

The practical consequence is that continued investment in alignment training does not progressively harden a model against jailbreak attacks. A frontier model can be simultaneously more strongly aligned and more susceptible to certain classes of adversarial attack than an earlier, less-aligned model, because the alignment optimizes for preference satisfaction in the training distribution while adversarial robustness requires a fundamentally different kind of worst-case resistance. The researchers describe these as distinct security properties that must be evaluated and defended separately.

Finding Two: Defenses Work Inconsistently — and Their Costs Are Real

The second finding is less surprising in direction but more alarming in scope: the defenses that do reduce attack success rates work inconsistently, and the costs they impose are high enough to create genuine enterprise viability problems.

Across the range of models, attack types, and system components the researchers evaluated, no defense worked reliably across all three dimensions. A technique that significantly reduced attack success rates against one model-attack combination frequently offered minimal protection against another. This fragmentation makes it difficult for security teams to know which defenses to trust across their actual deployment environment — which spans multiple models, multiple task types, and multiple configurations.

The more uncomfortable finding is what effective defenses cost. The study documents over-refusal and latency defense costs — the tendency for defended models to decline legitimate, benign requests that trigger their safety classifiers as a side effect of being tuned for adversarial inputs. For enterprise AI agents performing knowledge work, legal review, customer operations, or software development, an elevated over-refusal rate is not a theoretical annoyance — it is a direct reduction in the system's practical value. Add the latency penalties that more thorough output filtering and multi-pass safety checking impose at scale, and a security team faces a choice between meaningful protection with meaningful productivity costs, or leaner monitoring that leaves more of the attack surface open.

The security-utility-efficiency evaluation framework the researchers introduce is designed precisely to make this tradeoff visible and quantifiable: how much attack surface reduction does a given defense provide, at what cost in legitimate task completion, and at what cost in response latency? Current evaluation practice conflates these dimensions, making it difficult for organizations to make informed decisions about which defenses are worth deploying.

Finding Three: Safety Dashboards Can Show Clean When the Pipeline Is Not

The third finding is the one most directly relevant to enterprises operating AI agents in production today, and the one most likely to require fundamental changes to how AI security monitoring is structured.

Current safety infrastructure is almost entirely focused on what the researchers call final-response safety: monitoring, filtering, and evaluating what the agent ultimately outputs to the user or to the system reading its results. An agent that produces a final response that passes the safety filter registers as safe — regardless of what happened in the intermediate steps that produced that response.

In agentic AI architectures, those intermediate steps are substantial. Unlike a chatbot that takes a user prompt and returns a response, an agent operates through a five-layer agentic pipeline: a user-interaction layer that receives the initial request; a planning and reasoning layer that decomposes the request into sub-tasks and generates a sequence of actions; a memory layer that stores intermediate results and persistent context across steps; a tool-use layer that invokes external systems including file systems, APIs, browsers, and code execution environments; and an inter-agent communication layer that delegates sub-tasks to other AI models.

The paper's empirical results show that adversarial attacks can compromise the planning layer (causing the agent to pursue a different objective than intended), the memory layer (poisoning the stored context in ways that persist across sessions and surface only in later interactions), or the tool-use layer (manipulating which tools are invoked and how) — while generating a final response that appears clean to the monitoring system watching the output. A low final-response attack success rate can mask a high intermediate compromise rate.

This is structurally analogous to a failure mode that AI alignment researchers have long warned about under the name "deceptive alignment" — where models appear aligned during evaluation while pursuing a different goal in deployment. The mechanism is different: in intermediate pipeline compromise, the model itself is not being deceptive, but an attacker has manipulated the pipeline so that the layer the monitoring system watches (final output) does not reflect the state of the layers it is not watching (planning, memory, tool invocations). The result is the same: the safety dashboard shows green while the system is not safe.

An example makes the exposure concrete. A user-facing AI agent tasked with researching a topic and drafting a report retrieves documents from the web during its planning phase. An adversarial payload embedded in one of those documents — a form of indirect prompt injection — corrupts the agent's plan for the research task, causing it to invoke tools in unauthorized ways, exfiltrate data, or generate conclusions that are deliberately misleading. The final report it produces may be formatted normally and pass every content filter in the output pipeline. Nothing in the safety dashboard signals an incident.

Why Current Evaluation Methodology Cannot See This

The evaluation and certification protocols that AI companies and their enterprise customers use to verify AI safety were designed for a different architectural model than the one currently being deployed.

Standard AI safety red-teaming operates on a chatbot paradigm: present one harmful prompt, observe one response, grade the response in isolation. That methodology produces useful information about whether a model will refuse direct harmful requests. It produces no information about what the model will do when embedded in a multi-turn, multi-tool, multi-agent pipeline — because it never evaluates that context. A model can pass every single-turn safety benchmark and still participate in pipeline-level compromise in 100% of relevant agentic sessions, exactly as Copilot's workflow jailbreak research from the Alan Turing Institute documented in a specific coding-agent context in July 2026.

The implication for every safety certification currently in use is direct: if it was issued on the basis of prompt-level red-teaming — which most are — it is silent on the category of risk that agentic deployment creates. It does not measure and cannot speak to whether the planning layer is manipulation-resistant, whether the memory layer can be poisoned, or whether tool invocations are adversarially redirectable.

The EU AI Act, which brought its high-risk provisions into force on August 2, 2026, has begun to recognize this gap explicitly. The Act requires that agentic logging cover intermediate events relevant to risk identification and system monitoring rather than only final outputs. That requirement — intermediate-step logging rather than final-output logging — reflects precisely what the Mia et al. paper argues is necessary. The regulatory framework is beginning to catch up to where the research has been pointing. Most enterprise monitoring infrastructure has not.

How to Change Your Security Posture Before a Certification Tells You To

The Mia et al. paper stops short of proposing a complete defense architecture — it is, by design, a systematization and empirical study, not a solution specification. But its findings point clearly toward what a solution would need to include, and what enterprises can act on now without waiting for that solution to be fully built.

The first and most immediately actionable step is to stop treating output-layer safety metrics as an indicator of pipeline safety. A low attack success rate on final responses tells you whether the output filter is working; it does not tell you whether the planning, memory, or tool layers are secure. These are different questions with different answers, and conflating them produces exactly the false assurance the paper documents.

The second is to push vendors for session-level audit logs — traces of what the agent did at each pipeline step, not just what it ultimately said. Salesforce's AI Control Plane, announced September 11, 2026, and Zscaler's Agentic SOC, released September 9, explicitly offer per-action agent behavior observability rather than output-only monitoring. Enterprises evaluating new agent infrastructure should treat intermediate-step logging as a prerequisite, not a premium feature.

The third is to apply least-privilege principles to every tool the agent can invoke — restricting what the planning and tool layers can access to what a given task actually requires, so that a compromised planning step cannot invoke capabilities it should not have been given. An agent that can only read the documents it was asked to read, and can only write to the output it was asked to produce, provides an attacker far less surface to exploit even when the planning layer is successfully manipulated.

The research agenda the paper calls for — session-level safety evaluation that grades the full trajectory of an agentic interaction, not individual output turns — does not yet exist at scale. Building it is the work the AI security community needs to take up. The Mia et al. paper is the clearest statement yet of why that work cannot wait, and why the safety certifications issued before it was written cannot be assumed to cover the ground enterprises need them to cover.

Frequently Asked Questions

Does a model that passes standard AI safety evaluations mean it's safe for agentic deployment?

Not according to the Mia et al. research. Standard safety evaluations test models using a chatbot paradigm: one prompt, one response, graded in isolation. Agentic deployments operate through multi-step pipelines in which the model reasons, plans, uses tools, and stores intermediate results across multiple turns. A model that passes every single-turn safety evaluation can still be compromised at the planning, memory, or tool-use layer during an agentic session — and those compromises may never appear in the final response that the safety filter monitors. The paper's core finding is that these are separate security properties that must be separately evaluated and defended.

What is "intermediate pipeline compromise" in AI agents, and why does it matter?

Intermediate pipeline compromise refers to adversarial attacks that succeed at layers of the agentic pipeline other than the final output — specifically, at the planning and reasoning layer, the memory layer, or the tool-invocation layer. What makes it especially dangerous is that a successful compromise at these layers can remain invisible to monitoring systems that only evaluate what the agent ultimately outputs. The agent can produce a final response that passes every content filter while having been directed, mid-pipeline, to pursue an objective the attacker chose. For security teams running safety dashboards, the dashboard may show no alerts while an active compromise has already occurred.

What is the relationship between AI alignment training and adversarial robustness, and why does it matter for enterprises?

These are distinct properties that safety training techniques like reinforcement learning from human feedback and Constitutional AI improve independently. Alignment training optimizes for the model's behavior on a distribution of normal use — teaching it to produce preferred, safe responses in typical contexts. Adversarial robustness is about the worst-case: how the model behaves when inputs have been deliberately crafted to exploit its learned patterns. Mia et al. find empirically that these are orthogonal — a more strongly aligned model is not necessarily more robust against adversarial attacks. For enterprises, this means that a safety certification based on alignment benchmarks does not imply protection against the adversarial attacks that security teams need to be defending against in production.

What can enterprises do right now while session-level safety evaluation infrastructure is still being built?

Three concrete steps follow from the paper's findings. First, stop treating final-output safety scores as a proxy for pipeline security — they measure different things. Second, demand intermediate-step logging from AI agent vendors: you need traces of what the agent did at each pipeline stage, not just what it said. Third, apply strict least-privilege permissions to every tool an agent can invoke, so that a manipulated planning layer cannot reach capabilities beyond what the task actually requires. These steps do not fully close the gap the paper documents, but they reduce the attack surface and improve the visibility needed to detect and respond to a compromise when one occurs.