Transforming LLMs into Efficient Cross-Encoders via Knowledge Distillation for RAG Reranking

arXiv cs.CL · 18 hours ago

Researchers fine-tuned LLaMA 3 (8B) as a reranker for RAG pipelines using supervised fine-tuning and 4-bit quantization to replace traditional cross-encoders. The fine-tuned model achieved 21% improvement in answer correctness and 14-19% gains in other metrics while reducing inference costs. This approach enables LLMs to serve as efficient rerankers without the quadratic complexity of standard cross-encoders.