The next AI security incident may look like normal operation

Companies are pushing AI deeper into everyday operations, from fraud checks to customer decisions to systems that can fetch information and act across connected tools. That shift is happening faster than governance. Deloitte’s 2026 State of AI in the Enterprise found that 74% of surveyed organisations expect to be using AI agents at least moderately by 2027, while only around one in five said they had mature governance for agentic AI.

That gap is where the trouble starts. In 2025, researchers disclosed EchoLeak, a vulnerability in Microsoft 365 Copilot that let a specially crafted email steer Copilot through indirect prompt injection. Under certain conditions, it could expose data the user was already allowed to see. No click on a malicious link was required. Microsoft fixed it and said it had no evidence of customers being affected in the wild, but it has since used EchoLeak as a handy example of what goes wrong when AI systems work directly with enterprise data and content.

The nasty part is how normal the attack path looked. The email could be perfectly legitimate business mail for an employee and still contain instructions that influence the AI reading it. That means security teams have to think about more than accounts, infrastructure and malware. They need to know which sources can shape model behaviour, what data the system can reach and what happens after it produces an output.

That matters because the system’s authority can turn a small mistake into a business problem. I saw that in an AI fraud detection setup that flagged legitimate customer activity as suspicious. The model was running as designed, but the wrong call could still reject an order, cancel a transaction or trigger extra scrutiny. Once that output gets pulled into a wider workflow, the consequences can spread well beyond the model itself.

How much permission the system has determines how ugly the failure gets. The Open Worldwide Application Security Project calls this “excessive agency”: giving an AI more function, permission or autonomy than it needs. IBM’s 2026 study of 2,000 technology leaders gives a sense of scale, with respondents reporting an average of 54 AI-agent incidents in the previous year and 17% of those classed as high severity. Among the high-severity cases, 37% involved data exposure or security breaches and 33% led to cascading system failures.

The lesson is pretty simple. Inputs matter as much as permissions, and outputs matter as much as intent. A system that drafts internal text is not the same as one that can retrieve customer records, alter data or influence payments. If an organisation gives AI real authority, it also has to give it strict boundaries, human review where it counts and a way to stop a bad decision from marching deeper into the process.