AWS says its AI security agent fixes 89% of bugs — but there’s a catch
The New Stack Amanda Caswell
AWS says its new security agent can fix 89% of bug tasks. The catch: that score often missed the exact historical bug the test was built around.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS has made Continuum generally available, pitching it as an AI agent for code vulnerabilities that can search repositories, check results against an AWS setup, and open pull requests with suggested fixes. The headline sounds strong for a reason: the company says its benchmark run beat earlier public results and puts the system at the top of the test it chose to show off.
The benchmark is CyberGym-E2E, and AWS published its result on October 5, more than two weeks before launch. In that run, Continuum completed 819 of 920 tasks, which works out to an 89.0% end-to-end success rate. AWS says that was above the previous public high of 65.9%.
But the number that matters most is doing some work here. CyberGym-E2E gives an agent 90 minutes to find and repair a bug in one of 139 open-source projects, with no bug report, no proof of concept, no crash log and no existing patch. The agent has to discover the flaw, trigger it, and then write a fix that stops the crash without breaking the project’s tests. All of the tasks are memory-safety bugs in C or C++, which AWS itself says leaves out a lot of other languages and vulnerability types.
The benchmark is graded in stages, and that’s where the catch shows up. Continuum found and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept tests passing in 89.0%. But the final check asks whether the patch also fixes the historical vulnerability the benchmark picked, and Continuum did that in 37.8% of tasks. AWS says the previous public high there was 26.2%.
That gap is real, and AWS has a point about why it exists. A repository can contain multiple vulnerabilities, so fixing a different one is still useful work even if it misses the benchmark’s chosen bug. Still, the result leaves an obvious question hanging: how well does Continuum do when a team points it at one specific vulnerability and asks it to fix that exact problem?
AWS says it ran the tests itself under the benchmark’s rules, including limits on internet access, and checked that the system wasn’t just pulling public fixes. The company says Continuum finished 89% of tasks inside the 90-minute limit, and that number rises to 93.7% with more time. For now, though, these are AWS’s own results, with the benchmark and tools public but the claims still coming from one side.
My take — AI-written commentary, not fact-checked reporting
This is the usual AI security move: show the flashy score, then tuck the inconvenient part into the fine print. Fixing a random bug is useful; fixing the bug you were actually asked about is the job. The industry keeps celebrating benchmark theatre because it is much easier than building tools that hold up in the mess of real codebases.
Read more about this at: The New Stack