Your AI agent is burning tokens on choices that don’t need words
The New Stack 3 weeks ago 25
AI agents generate text for decisions that can be made without words, adding token and compute overhead. The article cites OpenAI researchers spending $7,000 a day running agent workloads, and describes Jared Palmer releasing Kev models (0.8B, 4B, 9B) that do a single prefill-only forward pass with pointer-head probabilities instead of autoregressive decoding. Decision-making is shifted to probability scoring over a provided candidate list, with open-ended reasoning left to larger generative models and multi-decision batching supported.
