Stop the token bleed: building token-efficient multi-agent systems
The New Stack Oladimeji Sowole
AI agents don’t get pricey because of one model call. They get expensive from all the extra retrieval, prompts, and rework around it.
Based on reporting by The New Stack, Oladimeji Sowole — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
Engineering teams shipping AI agents tend to find the same awkward truth: the model bill is often not the real problem. The real drag comes from everything wrapped around it — repeated retrievals, duplicated instructions, extra tool calls, bloated context windows, and multiple agents chewing on the same material. A small proof of concept can shrug that off. A production system handling thousands of requests a minute cannot.
That is why token efficiency is being treated here as a systems problem, not a prompt-writing trick. In a typical multi-agent flow, one agent pulls documents, another plans, another writes, another reviews, and each step may drag the same context through the pipeline again. By the time the answer comes back, the architecture may have burned through tens of thousands of unnecessary tokens. The fix starts before the model is even invoked.
The proposed setup puts cheaper logic in front of the expensive stuff. An intent router can send simple requests away from the LLM entirely. A semantic cache can return a prior answer when the same question hits the same documents. A context budget manager keeps retrieval from stuffing the window with more text than the model needs, and the article’s example uses a hard cap of 2,500 tokens. Retrieval also happens once, then gets shared across the downstream agents instead of being repeated.
Model choice matters too. The article argues for routing small jobs to a smaller model and saving the larger one for harder requests. It also pushes for token estimates before sending, structured output validation, and retries and timeouts configured through the environment. The point is less glamour, more discipline: production systems need to fail cleanly, stay predictable, and stop paying for work they don’t need to do.
The bigger lesson is blunt. Token waste is usually architectural waste. When teams measure it, the wins come from cutting duplicate retrieval, caching, routing simple work away from the model, and keeping context tight — not from shaving a few words off a prompt.
My take — AI-written commentary, not fact-checked reporting
This is the part of AI that doesn’t fit on a demo stage, which is probably why it matters. The industry still loves to talk about bigger models, while the boring adult work is to stop asking the same question three times and paying for the privilege. Open or closed barely matters if the system is built like a leaky bucket.
Read more about this at: The New Stack