Your agent is only as good as your infrastructure
The New Stack Selene Cecchinel
Your agent worked fine in tests, then slowed down in production. The model stayed the same; the infrastructure decided whether it felt fast or flaky.
Based on reporting by The New Stack, Selene Cecchinel — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
A pull-request agent can look brilliant in testing. It reads the diff, searches the codebase, runs tests, checks whether CI was already red, and drafts a comment before you’ve even finished reading the change yourself. Then production shows up and spoils the mood. The same workflow starts running behind every other PR review the team’s agents touch that hour, and suddenly some reviews finish in seconds while others sit for minutes because a test step landed on a busy node.
That is the core point here: the agent didn’t get worse. The execution environment did the deciding. For agentic applications, the workload is not a single prompt and a single response. It is a chain of inference calls and tool actions, and each link depends on the one before it. A chatbot usually handles one inference call per message. An agent can turn one message into search, retrieval, API calls, code execution, evaluation, and another round of reasoning before it answers.
That difference changes what latency means. In these multi-turn workflows, every step is sequential. If a database query takes two seconds, the next reasoning step cannot even start until that result comes back. The model may generate tokens quickly, but the rest of the chain drags. And if one link slows or fails, the user does not see an orchestration diagram. They see an agent that hung, or one that gave up.
The same pattern makes the bill harder to predict. Agent workloads pause while they wait on external systems, then resume in bursts when those systems return. That can leave GPUs idle one moment and slammed the next, with context sometimes evicted while the agent waits. If GPU use looks like a heartbeat — flat, then spiking when tool results land — the stack is probably being forced to handle a workload it was not built for.
The article’s answer is simple enough: infrastructure has to be built for the whole chain, not just the model call at the center of it. That means steady performance across long workflows, quick scaling when demand spikes, and costs that track actual usage instead of some polite average nobody lives in.
My take — AI-written commentary, not fact-checked reporting
This is the part AI teams keep pretending is unglamorous, which is exactly why it matters. Everyone loves to brag about the model and then act shocked when the plumbing turns the agent into a very expensive delay machine. The industry has spent years worshipping the demo; now it has to pay for the boring bits, which is rude but fair.
Read more about this at: The New Stack