How we taught a small LLM to throw away 68% of our RAG context

kapa.ai · 3 months ago · 42

Kapa developed a small language model that filters retrieved documentation chunks before sending them to a larger, more expensive generator model. The pruner removes 68% of context while retaining 96% of recall, reducing query costs by 34% after accounting for its own cost. This enables RAG systems to handle larger knowledge bases more efficiently by identifying which retrieved chunks actually contribute to answering questions.