Most AI Work Can Wait

Everyone building AI agents seems to start the same way: pick a flashy model, then bolt an architecture around it. The argument here flips that order entirely, and it's a good one. The model is supposed to be the last decision you make, not the first. What actually matters is the router — the unglamorous bit of code deciding which tier of model handles which request.

Get that router right, the piece argues, and 70 to 80 percent of your agent traffic can run on local models that cost basically nothing per call, or on async batch models that cut spend by more than 90 percent compared to real-time inference. Coinbase's Brian Armstrong made a similar point last week, noting the company halved its AI spend even as token usage kept climbing. His fix wasn't spending alerts or nagging engineers to be careful. It was better defaults, smarter routing, and caching.

The framework splits routing into three separate jobs, and conflating them is apparently where most teams go wrong. A skill classifier figures out what the task actually is — draft a reply, summarize a repo, run a migration. The router then decides which tier should execute that operation, looking not at the raw prompt but at features like complexity, context size, and how similar requests performed historically. Only after that does a model selector pick the cheapest option within that tier that clears a confidence bar. Bury the model choice inside the prompt itself, as many teams do, and you lose the ability to A/B test models against a consistent, well-defined operation.

The real unlock is realizing how little AI work actually needs to happen instantly. Local compute is nearly free, and async batch reasoning runs roughly two orders of magnitude cheaper than real-time. Once you accept that a draft reply, a repo summary, or a nightly evaluation run doesn't need to land in the next second, you can queue it — and queueing is what makes the cheap tiers usable at scale.

The author's own agent runtime now layers two feedback loops on top of the router. A synchronous predictor flags risky signals up front — missing context, long dependency chains, security-sensitive prompts — before a task even runs. A nightly batch evaluator then reviews the previous day's traces and adjusts the router's weights, itself running on cheap async inference so the feedback loop doesn't undo the savings it's meant to protect.