Building Cost-Efficient Enterprise RAG applications with Intel Gaudi 2 and Intel Xeon

Hugging Face Blog · 2 years ago

Intel demonstrated a retrieval-augmented generation application using Gaudi 2 accelerators and Xeon CPUs to answer questions about company financial documents. In benchmarks comparing Llama2-70B performance across four Intel Gaudi 2 and four Nvidia H100 systems with 16 concurrent users, Gaudi 2 delivered 2.27 times better cost efficiency despite 0.88x lower throughput. The results show enterprises can reduce total cost of ownership by deploying RAG systems on Intel platforms rather than competing hardware.