Arm and Google offer a smarter option to run agentic AI workloads
The New Stack Poornima Apte ● Covered by 2 sources
Google and Arm are pushing CPUs, not just GPUs, to run the boring-but-critical parts of AI agents. Turns out orchestrating agents on Axion chips can be up to 30% cheaper than rival clouds.
Based on reporting by The New Stack, Poornima Apte — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Everyone's obsessed with GPUs and TPUs when they talk about AI infrastructure, but Google and Arm are making a case that the real bottleneck for agentic AI is somewhere less glamorous: the orchestration layer. That's the stuff happening between the flashy model calls — routing tasks, managing memory, talking to APIs, deciding which tool to use next. According to Bhumik Patel, Arm's Director of Software Ecosystem Development, this is exactly the kind of distributed, concurrent work CPUs have always been good at, and throwing accelerators at it is overkill.
Google's answer is Axion, its custom Arm-based server chip that debuted in April 2024 as the company's first in-house CPU built specifically for cloud and AI-era workloads. It builds on more than ten years of Google's silicon work, which apparently makes it easier to fold customer feedback directly into chip design. The pitch here isn't that Axion replaces GPUs — it's that pairing Axion CPUs with specialized accelerators lets companies split the work intelligently: heavy reasoning and generation go to the expensive chips, while the constant, high-concurrency busywork of agent orchestration runs on CPUs built for exactly that.
The numbers Google is citing are specific enough to matter. Mo Farhat, Axion's Group Product Manager, says the GKE Agent Sandbox running on Axion N4A instances delivers up to 30% better price performance than the same workload on a competing hyperscale cloud. That sandbox is an open-source Kubernetes primitive built to safely run AI-generated code — a real concern once agents start writing and executing their own scripts autonomously. Letting untrusted code loose in a standard cluster is asking for trouble, since it could reach other applications or the node itself.
Google's fix leans on gVisor, its own application kernel, plus default-deny network policies and support for alternative isolation tools like Kata Containers. The system intercepts system calls before they touch the host kernel, isolating risky operations with sub-second latency. There's also a practical trick for cost control: GKE Pod snapshots let idle agents freeze their exact state and resume later, or move across nodes, without losing context — useful for long-running agents or reproducing a baseline environment for spinning up new sandboxes quickly.
The broader argument is really about economics. As agents multiply and generate more tokens with unpredictable bursts of activity, running everything on accelerator hardware gets expensive fast — what Google calls the inference tax. Splitting workloads between CPUs and accelerators, the companies argue, is how agentic AI stays financially sane at scale, not just technically feasible.
My take — AI-written commentary, not fact-checked reporting
This is basically Google and Arm admitting that the AI industry has been solving the wrong bottleneck for two years — everyone chasing GPU supply while the orchestration plumbing quietly ate the budget. I like that it's a genuinely boring, unsexy fix, because boring and cheap usually wins once the hype cools. Also worth noting: this is Arm's architecture doing real work in hyperscale AI infra, not just phones, which says something about where the open-vs-proprietary silicon fight is actually heading.
Read more about this at: The New Stack