Why CPUs still matter in the age of AI agents
The New Stack Frederic Lardinois ● Covered by 2 sources
AI agents are making CPUs matter again. They’re not just for chat; they’re now doing the control work that keeps agents moving safely.
Based on reporting by The New Stack, Frederic Lardinois — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
For years, AI infrastructure talk has mostly meant GPUs and TPUs. But Bhumik Patel of Arm and Mo Farhat of Google are arguing that the CPU is quietly moving back to the center of the story as AI shifts from chatbots to agents.
Farhat, who leads product management for Axion and Arm-based virtual machines at Google Compute Engine, compares the CPU to an air traffic controller. That’s the job now: not generating the flashy answer, but coordinating what comes next. Agents call tools, run code, and sometimes spin up environments for that code to execute. The orchestration around all of that is full of branching control flow, and that is exactly the sort of work CPUs are built for.
The model itself still usually runs on accelerators. But CPUs are doing a lot of the surrounding work: orchestration, data preparation, semantic search, and vector databases. Patel says Arm is focused on the software and infrastructure layers needed to make those workloads scale, and he points out that different kinds of agents are doing different kinds of code execution and API calling. Not everything needs a giant model either. Farhat says smaller models, including summarizers, recommenders, and evaluators, still have a role, and that six- to eight-billion-parameter models are doing much better than they used to. For some specialized jobs, he says, CPUs can hit roughly 25 tokens per second, which can be enough.
The harder problem is safety. If agents are writing and running code, they need sandboxes so they do not trash production systems. Patel says if the code is not trusted, the environment gets sandboxed. Google is pitching gVisor as the isolation layer, and Farhat is blunt about why that matters: the company operates in a zero-trust environment. Google’s GKE Agent Sandbox is meant to handle scale too, with Farhat saying it can spin up 300 sandboxes per second per cluster. Patel says pod snapshots and warm pools of suspended environments help avoid keeping everything fully provisioned.
Google is also trying to make the economics work. Farhat says Axion can deliver up to 2x the price performance of comparable current-generation virtual machines, along with over 60% better energy efficiency. C4A is aimed at consistently high performance, while N4A is pitched for more cost-optimized workloads. The bigger message is that this is not a CPU-vs-GPU story anymore. Farhat says we are in a fluid compute world, and customers should plan on using CPUs, GPUs, and TPUs together as agents spread.
My take — AI-written commentary, not fact-checked reporting
This is the rare AI infrastructure argument that is actually sane: not every problem needs a monster accelerator shoved at it until the bill screams. Agent systems are mostly coordination, trust, and plumbing, which is exactly where CPUs have always earned their keep. The industry loves shiny silicon, but the boring chip is often the one doing the real work.
Read more about this at: The New Stack