How we taught a small LLM to throw away 68% of our RAG context
kapa.ai
Kapa built a cheap LLM filter that sits between retrieval and generation, tossing out 68% of RAG context before the expensive model reads it. It keeps 96% recall and cuts query costs by a third — because rerank scores alone can't tell noise from a chunk that's only useful paired with another.
Based on reporting by kapa.ai — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
RAG has a quiet cost problem that nobody likes to talk about: the generator, the priciest model in the pipeline, is billed for every chunk of context it reads, including the ones it never actually uses. Kapa, which builds AI assistants over sprawling technical docs, found that retrieved chunks eat about two-thirds of the cost of a single query — more than the answer, the chat history, and the system prompt combined. Trim a chunk, save roughly 4% per query. Trim a lot of chunks, and the savings start to matter.
The obvious move is to just use the reranker's own scores as a cutoff. Kapa tried it and it fell apart for two reasons. Rerank scores are relative rankings, not calibrated measurements, so a 0.7 threshold means something different on every query. Worse, relevance often isn't a property of one chunk at all — in one production example, a chunk about audit logs only made sense paired with a second chunk that never mentioned audit logs by name. A pointwise reranker scores each chunk alone and has no way to see that the two belong together. Even a clever fix involving synthetic 'anchor' chunks planted at known relevance levels didn't help, because the underlying scores were still wrong, just wrong on a fixed scale.
So the team gave up on scores and brought in a second, cheap LLM to sit between the reranker and the generator. It reads the question and every retrieved chunk at once and grades each one on a five-point scale, from Essential to Unrelated, defined explicitly enough that a fixed cutoff actually holds across queries. Because the model sees the whole set together, it can catch chunks that are only useful in combination — the exact failure mode that killed the scoring approach.
The numbers are the interesting part. Plain top-N truncation, the naive baseline, can only shave off about 7% of chunks before recall drops to 98%. Kapa's listwise grading scheme gets to roughly 68% compression while still preserving 96% recall, and after accounting for the extra LLM call, the whole query gets about a third cheaper. It's not free: running a small model on every query in the critical path adds roughly 0.7 seconds of latency, and the generator's own speedup from reading less text barely makes a dent in that.
That tradeoff is why Kapa turned pruning on by default only for agentic use, where retrieval is one tool call among many and an extra fraction of a second barely registers next to everything else an agent already does. For latency-sensitive single-shot answers, it's left optional. The bigger lesson buried in here is less about RAG specifically and more about a pattern showing up everywhere in LLM pipelines: cheap models are increasingly useful not for generating the final answer, but for deciding what the expensive model is even allowed to see.
My take — AI-written commentary, not fact-checked reporting
This is a nice reminder that half the 'agents don't need RAG anymore' discourse ignores what happens once a knowledge base actually gets large and messy — retrieval isn't going away, it's just getting a cheap-model babysitter. What I like here is that Kapa didn't reach for a bigger model to fix the problem, they reached for a smaller, dumber one used more cleverly, which is the opposite of the industry's default instinct to throw flagship-tier compute at everything. More teams should be building these little judgment layers instead of assuming scale solves cost.
Read more about this at: kapa.ai