How we taught a small LLM to throw away 68% of our RAG context
TLDR Dev
Kapa developed a small language model that filters retrieved documentation chunks before sending them to a larger, more expensive generator model. The pruner removes 68% of context while retaining 96% of recall, reducing query costs by 34% after accounting for its own cost. This enables RAG systems to handle larger knowledge bases more efficiently by identifying which retrieved chunks actually contribute to answering questions.
Why it matters
A small, cost-effective LLM was implemented to prune retrieved context chunks for question-answering systems in order to improve efficiency, successfully discarding 68% of unnecessary chunks while maintaining 96% recall.
