CPU Optimized Embeddings with 🤗 Optimum Intel and fastRAG

Hugging Face Blog · 2 years ago

Hugging Face released optimized CPU-based embedding models using Optimum Intel and quantization techniques, targeting RAG pipeline performance improvements. The quantized BGE models achieved up to 4.5x latency speedup compared to original models while maintaining less than 1.55% accuracy loss on MTEB retrieval tasks. This enables faster document encoding and query processing on Intel Xeon CPUs without requiring specialized hardware.