Your AI agent is burning tokens on choices that don’t need words
The New Stack Amanda Caswell
AI agents keep using full text models for simple choices. Kev skips the wordy step, so routing and checks can get cheaper and faster.
Based on reporting by The New Stack, Amanda Caswell — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AI agents are wasting a lot of compute on something that doesn’t need prose at all. A routing decision, a safety check, a simple yes or no: these are choices, not essays. Yet many agents still hand them to a generative model, wait for text, then parse that text back into something useful. OpenAI’s own researchers recently said they were spending $7,000 a day on agent workloads, which gives the complaint some bite.
Kev is a different bet. It’s a new family of open decision models built on Qwen 3.5, and Jared Palmer released the latest version on Sunday in 0.8 billion, 4 billion, and 9 billion parameter sizes. Instead of decoding output token by token, Kev runs in a prefill-only mode: it takes the state, the question, and the candidates, then reads the answer from a pointer head in one forward pass. No autoregressive loop. No extra words.
The model supports three decision types — Noul for yes/no, Choice for picking among candidates, and Score for ordered levels — which matches TypeSafe’s System One API. Developers supply the state and the question, and Kev returns probabilities over the options it was given. For tool routing, that might look like search at 0.82, database at 0.13, calculator at 0.05. It can still be wrong, but it can’t invent a new option that wasn’t in the list.
That distinction matters because agent loops are full of these bounded calls. Routing, ranking, escalation, tool selection, safety checks — all of them can happen before the model writes anything user-facing. Palmer’s docs say the 4B model handled three questions in 277 milliseconds in bf16 on an M5, though that isn’t a controlled comparison against Qwen generating equivalent answers on the same hardware, so it doesn’t prove a speedup by itself. It does show the point of the design: keep the decision layer small and direct, leave the open-ended reasoning to a larger model.
The tradeoff is obvious. Kev-9B reached 83.7% accuracy on Palmer’s locked out-of-domain test, but Palmer also says the probabilities can drift on unseen source distributions, which makes threshold-based agent logic trickier. Fine-tuning can also hurt general-knowledge and arithmetic performance, especially in the smaller models. That’s fine if Kev stays in its lane. It’s less fine if someone expects it to be a magic brain in a box.
My take — AI-written commentary, not fact-checked reporting
This is the right kind of boring. Not every agent choice needs a poetry generator attached to it, and the industry’s habit of using a sledgehammer for a checkbox has been expensive theater. Open, local decision models make more sense here than another grand AI prophecy with extra tokens on top.
Read more about this at: The New Stack